Azure Application Gateways do not resolve Private Endpoints of storage accounts via custom DNS servers

Question

Good day,

I got a couple of setups looking like this:

• An Application Gateway ( WAF V2 or Standard V2) is placed in a VNET configured with custom DNS servers (They were restarted after the last change in this DNS servers list, so they are updated).
• These custom DNS servers are used to resolve addresses of Private Endpoints - for App Services and for static websites on Storage Accounts - and they resolve them successfully:
  appservicename.azurewebsites.net ==> appservicename.privatelink.azurewebsites.net ==> private IP
staticsitename.z6.web.core.windows.net ==> staticsitename.privatelink.web.core.windows.net ==> private IP
• Private Endpoints of both these types are defined as backend pools of Application Gateways.
• The Application Gateways work perfectly well with Private Endpoints of App Services. If I capture traffic on the DNS servers, I can see queries for these App Services names sent after the App Gateways are started. These names are resolved, health probes succeed and traffic is forwarded to these App Services. All's fine.
• However, the App Gateways don't even try to resolve names of Private Endpoints of Storage Accounts (for blob service or for static website service). I don't see any queries about their names arriving to our custom DNS servers. These names are resolved to their public IPs, and App Gateways reach these backends via another route - by Service Endpoints. So the App Gateways' subnet must be listed as allowed at the Firewalls and virtual networks page of these Storage Accounts, otherwise the health probes fail with error 404 (Private Endpoints shouldn't be affected by these settings).
• I made an experiment:
  a.) defined a Private DNS Zone for these Private Endpoints privatelink.web.core.windows.net and added to it relevant records resolving to private IPs;
  b.) bound it to the VNET where an Application Gateway resides;
  c.) stopped & started the Application Gateway;

Only after that the App Gateway resolved the staticsitename.z6.web.core.windows.net record to its Private Endpoint IP and was able to reach the Storage Account regardless of its Firewalls and virtual networks page settings.

So as I understand:

• Application Gateway uses custom DNS servers assigned to its VNET to resolve most of its backend servers FQDNs.
• However, for some of them (like Storage Accounts names under the windows.net domain) it uses the Azure built-in DNS service (168.63.129.16), ignoring custom DNS.
• So the only way to make an App Gateway resolve correctly addresses of Private Endpoints for Storage Accounts (and possibly other PaaS resources) is to attach a Private DNS Zone to its VNET, making the built-in Azure DNS service to read records from this Zone.

I didn't find anything like this in documentation.

Questions:

• Is this correct?
• If it is, what's the logic behind this way of splitting DNS queries? Is there any way to disable it?

Thanks,
Mucius.

Answer 1

@Cat Mucius

Your observations are correct; we pin a subset of FQDNs to always utilize Azure DNS (168.63.129.16 address). This is as per design due to Azure's internal architecture. As a workaround, customers can associate the privatelink DNS zone to the virtual network, regardless if they are utilizing custom DNS servers on the Vnet or not. This experience will be improved in an upcoming feature, targeted for the back half of this year. Hope this helps.

Please let us know if you have any further questions and we will be glad to assist you further. Thank you!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Answer 2

Yes, I know the architecture.
Moreover, as I mentioned, I tried binding a Private DNS Zone directly to the VNET hosting my Application Gateway - and it worked, the Storage Account's endpoint got resolved to its private IP.

My question is:

1. Why on earth would Application Gateway send some queries to my DNS forwarders configured for its VNET - and other queries to Azure DNS resolver 168.63.129.16? Usually a DNS client would send all queries to the same set of resolvers, regardless of a queried domain.
2. In what other cases would it do this? What is the guiding principle?

Unfortunately, this is not documented anywhere. I opened a ticket regarding the documentation - https://github.com/MicrosoftDocs/azure-docs/issues/87157 - but it seems nobody reacts to it.

Regards,
@Cat Mucius .

Answer 3

@Cat Mucius Whether it is App Service or Storage, private endpoint name resolution for Private Link resources can be done via Azure Private DNS or your DNS forwarder that forwards DNS queries to the recursive resolvers in Azure provided via the virtual IP 168.63.129.16 .
Please go through some of the best practices provided in the below link
https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/private-link-and-dns-integration-at-scale#private-link-and-dns-integration-in-hub-and-spoke-network-architectures
You can create private dns zones for the private link resources. The DNS Server Vnet can be linked to the private DNS Zones. The DNS servers use the Azure-provided DNS resolver (168.63.129.16) as a forwarder. All workload VNets can use the DNS servers hosted as the primary and secondary DNS servers.

Answer 4

Also do not forget to enable NETWORK POLICY FOR PRIVATE ENDPOINTS.
Even if you have dns resolution working, Without the network policy, route tables to private endpoints are ignored.