how to connect to AWS RDS Postgres from Azure data factory when Public accessibility is no in RDS

Question

how to connect to AWS RDS Postgres from Azure data factory when Public accessibility is no in RDS

Tharun CS 0

I have a AWS RDS Postgres , I need to fetch data AWS RDS Postgres and move it into AZure using Azure Data factory, I have setup VPN connectivity between AWS & Azure resource , the local gateway shows connection status as connected but i am not able to connect to AWS RDS postgres and the error reads as :"ERROR [08001] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Connection refused. Verify Host Name and Port Number. ERROR [HY000] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Can't connect to server on 'dev-dc-v2-db-16-02-2023.cogypllvhr9i.ap-south-1.rds.amazonaws.com' Activity ID: 56263991-4d5f-43da-a1a8-ba37f8586dc7." Please let me what is the best possible solution on this

KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-05T09:26:41.88+00:00

@Tharun CS

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

First we should isolate if this is an issue with Connectivity or AWS RDS Postgres/Azure Data factory misconfiguration.

Please spin up a VM in Azure and a VM in AWS and check if they have connectivity between them.

Cheers,

Kapil.
KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-08T06:57:41.73+00:00

@Tharun CS

May I know if you got a chance to review my previous comment?

Please let me know if you are facing any challenges or if there are any follow-up questions, I shall be glad to address them.

Thanks,

Kapil
Tharun CS 0 Reputation points

2023-05-08T08:38:30.43+00:00

I need to have a VM in Azure to check the connectivity? I have a VPN site to site connection where within local gateway the Ip address from tunnel (AWS) shows as connected . my question is to connect to a adf within a resource where Virtual network & Local gateway is configured to allow connection to fetch data from RDS
KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-09T05:01:02.9666667+00:00
@Tharun CS

I need to have a VM in Azure to check the connectivity?

Yes

I have a VPN site to site connection where within local gateway the Ip address from tunnel (AWS) shows as connected

I understand, but this does not mean traffic flow is actually happening and connections are indeed allowed.

It is highly possible that there might be firewalls in Azure or AWS blocking the traffic or there is a Traffic Selector Mismatch.

my question is to connect to a adf within a resource where Virtual network & Local gateway is configured to allow connection to fetch data from RDS

Again, this would require expertise from ADF and RDS experts.

VPN connection is different and ADF is totally different.

Firstly, we have to make sure the VPN connection is working and for that, we need to have a VM in Azure and VM in AWS and check if they can ping each other.

Only if the above works, we can check the configuration of ADF

Cheers,

Kapil
KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-15T04:19:33.84+00:00

@Tharun CS

Can you please update us if the action plan provided was helpful?

Should there be any follow-up questions or concerns, please let us know and we shall try to address them.

Thanks,

Kapil

1 answer

Your answer

KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-05T09:26:41.88+00:00

@Tharun CS

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

First we should isolate if this is an issue with Connectivity or AWS RDS Postgres/Azure Data factory misconfiguration.

Please spin up a VM in Azure and a VM in AWS and check if they have connectivity between them.

Cheers,

Kapil.
KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-08T06:57:41.73+00:00

@Tharun CS

May I know if you got a chance to review my previous comment?

Please let me know if you are facing any challenges or if there are any follow-up questions, I shall be glad to address them.

Thanks,

Kapil
Tharun CS 0 Reputation points

2023-05-08T08:38:30.43+00:00

I need to have a VM in Azure to check the connectivity? I have a VPN site to site connection where within local gateway the Ip address from tunnel (AWS) shows as connected . my question is to connect to a adf within a resource where Virtual network & Local gateway is configured to allow connection to fetch data from RDS
KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-09T05:01:02.9666667+00:00

@Tharun CS

I need to have a VM in Azure to check the connectivity?

Yes

I have a VPN site to site connection where within local gateway the Ip address from tunnel (AWS) shows as connected

I understand, but this does not mean traffic flow is actually happening and connections are indeed allowed.

It is highly possible that there might be firewalls in Azure or AWS blocking the traffic or there is a Traffic Selector Mismatch.

my question is to connect to a adf within a resource where Virtual network & Local gateway is configured to allow connection to fetch data from RDS

Again, this would require expertise from ADF and RDS experts.

VPN connection is different and ADF is totally different.

Firstly, we have to make sure the VPN connection is working and for that, we need to have a VM in Azure and VM in AWS and check if they can ping each other.

Only if the above works, we can check the configuration of ADF

Cheers,

Kapil
KapilAnanth-MSFT 49,536 Reputation points Microsoft Employee Moderator

2023-05-15T04:19:33.84+00:00

@Tharun CS

Can you please update us if the action plan provided was helpful?

Should there be any follow-up questions or concerns, please let us know and we shall try to address them.

Thanks,

Kapil

Answer 1

To connect to an AWS RDS Postgres instance from Azure Data Factory when public accessibility is disabled, you need to ensure that the network connectivity between Azure and AWS is correctly configured, and the security groups and firewall settings allow traffic between the two environments. Here are a few steps to troubleshoot your issue:

Check network connectivity: Make sure that the VPN connection between your Azure and AWS environments is correctly configured and active. You can verify this in the AWS VPC console for the VPN connection status and the Azure portal for the Azure Virtual Network Gateway connection status.
Check AWS security group rules: Ensure that the security group associated with the RDS instance allows inbound traffic from the Azure Data Factory's managed private endpoint or from the subnet where your integration runtime is running in Azure.
Check Azure NSG rules: If you have a Network Security Group (NSG) associated with the Azure subnet, make sure it allows outbound traffic to the AWS RDS Postgres instance.
Verify the connection string: Double-check the connection string used in the Azure Data Factory's source settings. Make sure the hostname, port number, and credentials are correct.
Test the connectivity: You can use tools like psql or an application like pgAdmin to test the connectivity from a VM in your Azure environment to the AWS RDS Postgres instance. This can help you identify if the issue is related to the Azure Data Factory configuration or the network settings.

If you have followed these steps and are still facing issues, you may need to review your network configuration in more detail or consult your network administrators to ensure proper connectivity between the Azure and AWS environments.

Share via

how to connect to AWS RDS Postgres from Azure data factory when Public accessibility is no in RDS

1 answer

Your answer