Connectivity issues between AKS and PostgreSQL Flexible Server

Question

Connectivity issues between AKS and PostgreSQL Flexible Server

Paul Breton 81

Hi all,

I have a complex problem to diagnose and solve. I have an AKS with pods connecting to an Azure Database for PostgreSQL Flexible Server. I deployed it first on a test environment and everything worked well. Then, in the production environment "everything" went wrong :

Pods are regularly killed by Kubernetes because their Healthcheck is taking too much time to answer
Pods are Spring boot Applications, just trying to connect to their PostgreSQL database on startup
Connections are using a Hikari pool of roughly (by default) ~10 sessions.
The healthcheck triggers a communication to ensure that the pod can connect to the PostgreSQL Server, which fails after ~30 secs (timeout), randomly (sometimes it does not fail, but it always fails within ~5-10 mins)
Because many different applications are using the same PostgreSQL Server, it actually hosts around 180~ sessions.

I tried many different things to diagnose the problem, and the conclusion always remains the same : connecting from the pods in this AKS ends in timeout (even manually with psql), whereas connecting from outside the pod works perfectly.

The main track I have is that the AKS and the PostgreSQL Server are not on the same Availability Zone (but on the same Region though, West Europe). AKS is not configured with a precise Zone, but PostgreSQL Flexible is configured to be on the AZ number 2. During my tests, I found that my AKS could correctly discuss with another PostgreSQL Flexible which was on AZ 3 (from another environment, the test environment).

Is there a possibility where there would be connectivity issues between Azure services located in two different Availability Zones ? Would it be a solution to migrate my PostgreSQL Server to the AZ number 3 ?

Thank you for your help,

shiva patpi 13,366 Reputation points Microsoft Employee Moderator

2023-08-22T15:52:20.28+00:00

@Paul Breton

Did you observe any difference between your test & Prod environment ? Was your testing environment AKS was also in the same AZ ? Which version of AKS is this ?

If many different applications are making use of the same PostGRE SQL and you said it times out - probably try testing out with a sample test PostGRE SQL equivalent to the problematic one and see how it goes ?

Regards,

Shiva.
Carlos Villagomez 1,106 Reputation points Microsoft Employee Moderator

2023-08-22T22:53:51.7033333+00:00

@Paul Breton

To add to @shiva patpi can you please clarify whare the exact differences between test and prod? For example, is it only the AV Zone? Or are there any others differences in configuration such as the VM SKU size, Disk type, control plane SKU, cluster add-ons, etc.

This will help us ultimately determine what is the root cause based on the observed differences between the test and prod environments.

Best,
Carlos
Paul Breton 81 Reputation points

2023-08-23T09:21:36.92+00:00
@Carlos Villagomez They are on two different tenants, and so in two different subscriptions (the two subscription do not have access to the same AZ subset, apparently). We deploy it using Terraform and configuration files which ensure that they are almost exactly the same. The only difference is the agents sizes : "Standard_D2_v3" in test and "Standard_DS4_v2" in prod.

@shiva patpi Except the one listed here, no. My AKS on the test environment also has the configuration set on "no preference" concerning the AZ. AKS version is 1.25.5. Concerning PostgreSQL, I tested different configurations with the 4 components : AKS test (no AZ), AKS prod (no AZ), PG test (AZ 3) and PG prod (AZ 2).

AKS test -> PG test : works perfectly

AKS test -> PG prod : works perfectly

AKS prod -> PG test : works perfectly

AKS prod -> PG prod : works but with latency, timeouts and error.

We did not spawn another PG on the prod environment and the same AZ as we assumed the results would be the same. As the solution is entirely Managed and that we just spawned it using the default configuration, why would there be any difference ?
Emmanuel IK 0 Reputation points

2023-10-26T10:11:19.37+00:00

Hi Please having same issue and was wondering if a solution to this was found. All seemed to have worked in a dev environment but now moved to a production environment and with the "Allow public access from any azure services" enabled on the postgres flexible server, the AKS pods still cant seem to reach the posgres server. Need to know what the solution for this is
Paul Breton 81 Reputation points

2023-10-26T10:14:30.84+00:00

Our problem was SNAT ports exhaustion. We had pods that were at the source of this problem, on the network layer of our AKS. So the problem was on our side, in our apps.

Accepted answer

1 additional answer

Your answer

shiva patpi 13,366 Reputation points Microsoft Employee Moderator

2023-08-22T15:52:20.28+00:00

@Paul Breton

Did you observe any difference between your test & Prod environment ? Was your testing environment AKS was also in the same AZ ? Which version of AKS is this ?

If many different applications are making use of the same PostGRE SQL and you said it times out - probably try testing out with a sample test PostGRE SQL equivalent to the problematic one and see how it goes ?

Regards,

Shiva.
Carlos Villagomez 1,106 Reputation points Microsoft Employee Moderator

2023-08-22T22:53:51.7033333+00:00

@Paul Breton

To add to @shiva patpi can you please clarify whare the exact differences between test and prod? For example, is it only the AV Zone? Or are there any others differences in configuration such as the VM SKU size, Disk type, control plane SKU, cluster add-ons, etc.

This will help us ultimately determine what is the root cause based on the observed differences between the test and prod environments.

Best,
Carlos
Paul Breton 81 Reputation points

2023-08-23T09:21:36.92+00:00

@Carlos Villagomez They are on two different tenants, and so in two different subscriptions (the two subscription do not have access to the same AZ subset, apparently). We deploy it using Terraform and configuration files which ensure that they are almost exactly the same. The only difference is the agents sizes : "Standard_D2_v3" in test and "Standard_DS4_v2" in prod.

@shiva patpi Except the one listed here, no. My AKS on the test environment also has the configuration set on "no preference" concerning the AZ. AKS version is 1.25.5. Concerning PostgreSQL, I tested different configurations with the 4 components : AKS test (no AZ), AKS prod (no AZ), PG test (AZ 3) and PG prod (AZ 2).

AKS test -> PG test : works perfectly

AKS test -> PG prod : works perfectly

AKS prod -> PG test : works perfectly

AKS prod -> PG prod : works but with latency, timeouts and error.

We did not spawn another PG on the prod environment and the same AZ as we assumed the results would be the same. As the solution is entirely Managed and that we just spawned it using the default configuration, why would there be any difference ?
Emmanuel IK 0 Reputation points

2023-10-26T10:11:19.37+00:00

Hi Please having same issue and was wondering if a solution to this was found. All seemed to have worked in a dev environment but now moved to a production environment and with the "Allow public access from any azure services" enabled on the postgres flexible server, the AKS pods still cant seem to reach the posgres server. Need to know what the solution for this is
Paul Breton 81 Reputation points

2023-10-26T10:14:30.84+00:00

Our problem was SNAT ports exhaustion. We had pods that were at the source of this problem, on the network layer of our AKS. So the problem was on our side, in our apps.

Answer 1

@Paul Breton
How do I troubleshoot connectivity issues between AKS and PostgreSQL Flexible Server?
After checking the details the best will be to open a ticket with Support.

There are many elements to be reviewed for this type of integrations starting by

Connection Pooling
If the server is PostgreSQL - Single Server this in under the deprecation period and would be better to use Azure Database for PostgreSQL - Flexible Serve
AV Zone should not be a meningful sittuation. Configurations
Is necessary to review the error messages in PostgreSQL and AKS., as the timeouts can be triggered from any of the related resources.
Is necessary to keep present latest Kubernetes versions are using cgroups v2 and exist limitations with JAVA versions
Subnet DNS and CoreDNS resolution

https://supportability.visualstudio.com/AzureDBPostgreSQL/_wiki/wikis/AzureDBPostgreSQL/503215/How-to-check-connectivity-errors

Grant Egress communication
https://learn.microsoft.com/en-us/azure/aks/outbound-rules-control-egress
Troubleshoot connection issues to pods or services within an AKS cluster (internal traffic)
https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/troubleshoot-connection-pods-services-same-cluster
How to check connectivity errors
https://supportability.visualstudio.com/AzureDBPostgreSQL/_wiki/wikis/AzureDBPostgreSQL/503215/How-to-check-connectivity-errors

https://supportability.visualstudio.com/AzureContainers/_wiki/wikis/Containers%20Wiki/540934/How-to-troubleshoot-connectivity-issues-with-databases-from-AKS-clusters

If it is possible to get the error messages could be a good point for starting the investigation.

Kubectl get events
Kubectl describe pods
Kubectl describe enpoints
etc.
Only difference is the agents sizes : "Standard_D2_v3" in test and "Standard_DS4_v2" in prod.
Concerning PostgreSQL, they tested different configurations with the 4 components : AKS test (no AZ), AKS prod (no AZ), PG test (AZ 3) and PG prod (AZ 2).
AKS test -> PG test : works perfectly
AKS test -> PG prod : works perfectly
AKS prod -> PG test : works perfectly AKS prod -> PG prod : works but with latency, timeouts and error.

To further troubleshoot this issue we're going to need to look at your resources in more detail. Please email the following to AzCommunity@microsoft.com and we'll get back to you promptly:

Subject: "Attn: kobulloc - Additional support required"
Email body: Your Subscription ID
Email body: A link to this thread so we can validate and expedite the request

If you don't receive a response within 24 hours, please reply to the thread so we can investigate.

Paul Breton 81 Reputation points

2023-08-31T11:04:25.0666667+00:00

Update 2 : the Azure team helped us to diagnose a problem concerning our a SNAT ports exhaustion, causing all connections to fail regularly as the Database was the main destination of our connections.

Thank you all !
Carlos Villagomez 1,106 Reputation points Microsoft Employee Moderator

2023-08-31T15:28:28.5266667+00:00

@Paul Breton Thanks for the update and glad to hear the issue has been successfully resolved! :)

Answer 2

Paul Breton 81

Update : I observed that these problems did not happen for PostgreSQL Flexible Servers on AZ 1. I was planning to migrate my server from AZ 2 to AZ 1 using High Availability and Planned Failover. Problem is, even this fails. I cannot enable High Availability on AZ 1 because it fails with a timeout error ! Exactly the same as when I try to connect to it from the AKS. I tried many times, and it always has the same result after 2 hours trying to create the replica : timeout error.

Carlos Villagomez 1,106 Reputation points Microsoft Employee Moderator

2023-08-24T18:34:18.09+00:00
@Paul Breton
Thank you for clarifying this issue. The failure to move from AZ2 to AZ1 indicates that either there is something wrong with the resource in AZ2 (most likely) or that this is an unsupported operation (less likely). As long as we rule this out as a general connectivity issue, then it seems like this may be a resource issue.

To further troubleshoot this issue we're going to need to look at your resources in more detail. Please email the following to AzCommunity@microsoft.com and we'll get back to you promptly:

Subject: "Attn: kobulloc - Additional support required"

Email body: Your Subscription ID

Email body: A link to this thread so we can validate and expedite the request

If you don't receive a response within 24 hours, please reply to the thread so we can investigate.
Paul Breton 81 Reputation points

2023-08-25T15:06:14.24+00:00

Fortunately, I received an answer. As another update, I wish to modify my previous statement about my tests. I tests the High Availability on AZ 3, and it was not working (failing with timeout after ~2 hours). It only is on AZ 1 that this feature works. So I enabled it on AZ 1 (from AZ 2) and plan on doing a failover to migrate my PostgreSQL database on AZ 1. However I am not certain of anything and if this will solve my problem.

Thank you for your help and I'll keep this thread updated with the support answers !
Carlos Villagomez 1,106 Reputation points Microsoft Employee Moderator

2023-08-25T16:51:40.5933333+00:00

@Paul Breton
Thank you for the update, and I'm very glad to hear that you have received an answer. Please do keep us updated here and once the issue is entirely resolved we will be sure to update the resolution here to help anyone else in the future who may face this issue.

Best,

Carlos V.
Carlos Villagomez 1,106 Reputation points Microsoft Employee Moderator

2023-08-30T17:55:16.51+00:00

Hello @Paul Breton

I hope this has been helpful! Your feedback is important so please take a moment to accept answers.](https://docs.microsoft.com/en-us/answers/support/accepted-answers)."https://docs.microsoft.com/en-us/answers/support/accepted-answers).")

If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!

Also, if you can please keep us updated once the case has been resolved and share in this thread to others in the Community who may face the same issue in the future for visibility that would also be appreciated.

Thanks!
Carlos V.

Share via

Connectivity issues between AKS and PostgreSQL Flexible Server

1 additional answer

Your answer