Azure Database PostgreSQL private access (virtual network integration) DNS resolution issue

Question

My team is currently running into a very specific issue with several Azure Database PostgreSQL servers, configured using private access (virtual network integration), and DNS resolution and connectivity to those servers from within several AKS clusters.

Everything in our environment is vnet integrated and using private networking and private DNS zones. From within our AKS clusters we have no other issues resolving other resources using private DNS zones, public services, or other services internal to our network.

For some some reason we cannot resolve DNS nor establish a connection to any of 5 different Azure Database PostgreSQL we have deployed. In 4 different AKS clusters we've deployed several services which attempt to connect to the servers and fail. For troubleshooting purposes I've tested DNS resolution using nslookup and postgresql connectivity using the psql client, from a Ubuntu pod/container, all of which fail with error message such as: "server can't find <record>.privatelink.postgres.database.azure.com: NXDOMAIN. These same servers can be resolved and connected to from anywhere else on our network including other Azure services (VMs, function apps, etc) as well as end user devices (laptops).

After hours of troubleshooting I found that deploying a PostgreSQL server using Private Endpoint rather than virtual network integration experiences none of these issues. Using Private Endpoint we can resolve and connect to the PostgreSQL server with no issues from both inside of the AKS clusters and outside.

It seems that there's a very specific issue with PostgreSQL servers using virtual networking integration and AKS clusters connectivity. Unfortunately these PostgreSQL servers are already being utilized for production purposes and before even thinking about migrating databases to new servers using Private Endpoints I wanted to post here for assistance, to check if there are known issues with PostgreSQL connectivity when using virtual network integration and more specifically from within AKS clusters.

Adding on that we experience these issues regardless of whether the AKS cluster is within the same vnet as the postgresql server or not. So we experience the same connectivity failures even with a postgresql server in the same vnet as the AKS cluster.

Answer 1

It seems like you're encountering a DNS resolution issue specifically related to Azure Database for PostgreSQL servers configured with private access (virtual network integration) from within your AKS clusters. Here are a few points to consider based on the context provided:

1. Private DNS Zone: Ensure that the Private DNS zone for your Azure Database for PostgreSQL is correctly linked to the virtual network where your AKS clusters are deployed. If the Private DNS zone is not linked, clients in the AKS clusters will not be able to resolve the database server's FQDN.
2. DNS Forwarder: If you're using a custom DNS server, make sure you have a DNS forwarder configured to resolve the FQDN of your Azure Database for PostgreSQL instance. The forwarder IP address should be 168.63.129.16.
3. Network Configuration: Confirm that there are no network security group (NSG) rules or other firewall settings that might be blocking traffic from your AKS clusters to the PostgreSQL servers.
4. Virtual Network Integration Limitations: Be aware of the limitations of virtual network integration. For example, once an Azure Database for PostgreSQL instance is deployed to a virtual network and subnet, it cannot be moved to another virtual network or subnet. This could affect your ability to change configurations if needed.
5. Testing Connectivity: Since you mentioned that connectivity works fine with Private Endpoint, it may be worth testing the connectivity from a VM in the same virtual network as the PostgreSQL server to isolate whether the issue is specific to AKS or a broader network configuration issue.

If the above checks do not resolve the issue, consider reaching out to Azure support for more in-depth troubleshooting, as they may have insights into specific configurations or known issues affecting your setup.

---

References:

• Network with private access (virtual network integration) for Azure Database for PostgreSQL

Answer 2

Hi @Roccuzzo, Matt

Thank you for contacting to Microsoft QA.

It looks like you're facing a pretty complex DNS resolution issue with your Azure Database PostgreSQL servers when accessing them from your AKS clusters. Here's a detailed set of steps and checks you can follow that should help you troubleshoot this problem:

Check Private DNS Zone Configuration:

• Ensure that the private DNS zone is correctly linked to the virtual network of both your PostgreSQL servers and your AKS clusters. You can do this by navigating to the Private DNS zone in the Azure Portal and checking for virtual network links.

Verify A Records:

  - Within your Private DNS zone, verify that the necessary A records exist for your PostgreSQL servers. If any records are missing, you will need to create A records mapping the PostgreSQL server names to their respective private IP addresses.
  
  **DNS Forwarding**:
  
     - If you are using a custom DNS server within your environment, make sure it is configured to forward queries for Azure services to the Azure DNS IP (168.63.129.16). This is crucial for the resolution of private endpoints.
     
     **Check AKS Network Configuration**:
     
        - Ensure that your AKS clusters have appropriate network policies that allow outbound traffic to the PostgreSQL servers. Any Network Security Groups (NSGs) should be checked to confirm they are not blocking necessary traffic.
        
        **Use Fully Qualified Domain Names (FQDN)**:
        
           - Always use the FQDN for connecting to your PostgreSQL servers (`<server-name>.privatelink.postgres.database.azure.com`). Avoid using IP addresses for connections, as they may change and can lead to resolution problems.
           
           **Flush DNS Cache**:
           
              - Sometimes DNS resolution issues can be local to the pod/container. You can flush the DNS cache within your AKS clusters or restart the pods to ensure they obtain the latest DNS records.
              
              **Run nslookup from Pod**:
              
                 - Execute a DNS query from a pod in the AKS cluster using `nslookup <your-server-FQDN>` to see if it resolves correctly. If it returns NXDOMAIN, you will need to revisit your DNS configurations.
                 
                 **Consult Your Networking Team**:
                 
                    - If the issue persists, it might be beneficial to collaborate with your networking or DNS team to validate the overall configuration.
                    

Since you're also experiencing this issue across multiple AKS clusters and regardless of whether they're in the same virtual network as the PostgreSQL instances or not, it strongly suggests that the problem lies within the DNS or networking configuration rather than the databases themselves.

Here are some follow-up questions that might help clarify the scenario further:

• Are there any custom DNS servers in use, or are you relying on Azure's DNS services?
• Have you recently made any changes in your VNet or AKS network configurations?
• Can you successfully resolve other resources using the same DNS configurations from the AKS clusters, excluding the PostgreSQL servers?

Hopefully, these steps help you get to the root of the issue! If you need more assistance, feel free to ask.

References:

• Private access with virtual network integration
• Using a Private DNS zone
• Verifying DNS Configuration
• DNS Troubleshooting in Azure
• Network-to-Network Connections