Azure Private Link with ARM or Bicep - Private Dns Zone Group does not create A records

Question

Hi,

I seem to run into an issue when deploying a private endpoint for Azure Event Hubs or Azure Redis Cache (on the same tenant and subscription).
I'm automating the deployment with bicep templates (see below), and I deploy a "Private Dns Zone Group", as advised in the tutorial https://learn.microsoft.com/en-us/azure/private-link/create-private-endpoint-template.
However when the deployment is done (successful), there is no record in the private DNS zone, so applications in the Vnet cannot resolve the service's private link domain (e.g. privatelink.redis.cache.windows.net). I used az network private-endpoint dns-zone-group list to see if the zone's status is correct:

[  
  {  
    "etag": "W/\"75028c29-638c-444a-b5e9-260eeded5a48\"",  
    "id": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_ID/providers/Microsoft.Network/privateEndpoints/pe-redis-cache/privateDnsZoneGroups/default-zone-group",  
    "name": "default-zone-group",  
    "privateDnsZoneConfigs": [  
      {  
        "etag": "W/\"75028c29-638c-444a-b5e9-260eeded5a48\"",  
        "id": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_ID/providers/Microsoft.Network/privateEndpoints/pe-redis-cache/privateDnsZoneGroups/default-zone-group/privateDnsZoneConfigs/pe-redis-cache-dns-zone-group-config",  
        "name": "pe-redis-cache-dns-zone-group-config",  
        "privateDnsZoneId": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_ID/providers/Microsoft.Network/privateDnsZones/privatelink.redis.cache.windows.net",  
        "recordSets": [  
          {  
            "fqdn": "redis-internal-blabla.privatelink.redis.cache.windows.net",  
            "ipAddresses": [  
              "11.2.0.5"  
            ],  
            "provisioningState": "Succeeded",  
            "recordSetName": "redis-internal-blabla",  
            "recordType": "A",  
            "ttl": 10  
          }  
        ],  
        "resourceGroup": "RESOURCE_GROUP_ID",  
        "type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups/privateDnsZoneConfigs"  
      }  
    ],  
    "provisioningState": "Succeeded",  
    "resourceGroup": "RESOURCE_GROUP_ID",  
    "type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups"  
  }  
]  

Everything seems fine but there is still no record. When I create the endpoint or even the DNS config manually via the portal, a record is correctly created. I checked the automation template suggested after the manual creation, and it's fundamentally the same as my bicep template.

Is there something I am missing? Should I also manually create the A record in the DNS zone?

Bicep template:

@minLength(1)  
param privateEndpointsSubnetId string  
  
@minLength(1)  
param privateEndpointName string  
  
@minLength(1)  
param targetPrivateLinkResouceId string  
  
@minLength(1)  
//already created  
param privateDnsZoneId string  
  
@allowed([  
  'redisCache'  
  'namespace' // (event hub namespace)  
])  
@description('See https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns#azure-services-dns-zone-configuration for the list of subresources. Make sure it matches the target resource.')  
param targetSubResource string  
  
@description('Tags to add to resources deployed by this template')  
param commonTags object  
  
resource privateEndpoint 'Microsoft.Network/privateEndpoints@2021-02-01' = {  
  name: privateEndpointName  
  location: resourceGroup().location  
  properties: {  
    subnet: {  
      id: privateEndpointsSubnetId  
    }  
    privateLinkServiceConnections: [  
      {  
        name: privateEndpointName  
        properties: {  
          privateLinkServiceId: targetPrivateLinkResouceId  
          groupIds: [  
            targetSubResource  
          ]  
        }  
      }  
    ]  
  }  
  tags: commonTags  
}  
  
resource privateDnsZoneGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2021-02-01' = {  
  name: 'default-zone-group'  
  parent: privateEndpoint  
  properties: {  
    privateDnsZoneConfigs: [  
      {  
        name: '${privateEndpointName}-dns-zone-group-config'  
        properties: {  
          privateDnsZoneId: privateDnsZoneId  
        }  
      }  
    ]  
  }  
}  

Answer 1

I want to confirm that I have seen exactly the same behaviour, at present, for Storage Account and Key Vault resources.

If I go in Portal to <Resource>->Networking->Private endpoint connections->+Private endpoint and run through the wizard I get a functioning A record in the private DNS zone.
If I export the template for that deployment and run it I do not.

There is only one obvious difference that I can see between the two deployment styles in terms of the deployed resources:
Click-Ops - Going to the Private Endpoint resource -> DNS Configuration

Bicep - Going to the Private Endpoint resource -> DNS Configuration

There is a customDNSConfig property in the Bicep template for private endpoints that doesn't get populated when you export. I've tried setting this with the expected fqdn of the service (and not an IP address) and it doesn't change the outcome. We can't populate the fqdn AND IP address because the NIC may not even exist when the private endpoint Bicep resource is deployed. There is definitely something going wrong with the Bicep resource creation for this type...

Answer 2

My colleague just had the same issue. According to the Private DNS zone -> Activity log there were actions of Delete record of type A
Event initiated byAzure Traffic Manager and DNS. Meaning Azure removed them for some reason. IaC deployment was incrimental.

The issue was solved by manually deleting Private endpoint & Nic and just deploy the same thing again.

Not a clean solution that's recommended for Production where you want up-time, for obvious reasons.
We suspect the reason for this is an Azure bug

Answer 3

Hello @Rémi Sormain ,

I checked your Bicep template and also did a bit of research on this and looks like "Microsoft.Network privateDnsZones/virtualNetworkLinks" is missing from your template. After creating the private DNS zone for Event Hubs domain, you need to create an association link with the virtual network.
Could you please add the same and try again?

Refer : https://learn.microsoft.com/en-us/azure/templates/microsoft.network/privatednszones/virtualnetworklinks?tabs=bicep
https://learn.microsoft.com/en-us/azure/event-hubs/private-link-service#configure-the-private-dns-zone
https://stackoverflow.com/questions/64342793/private-endpoint-for-a-storage-queue-in-arm

Kindly let us know if the above helps or you need further assistance on this issue.

----------------------------------------------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 4

Thanks for your answer @GitaraniSharma-MSFT .
The private dns zone is created in another module, and the virtual network link is already configured. Here is the template used to create the DNS zone

@minLength(1)  
param privateDnsZoneName string  
  
@minLength(1)  
@description('The vnet to connect with the private DNS zone')  
param vnetId string  
  
@description('Tags to add to resources deployed by this template')  
param commonTags object  
  
resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {  
  name: privateDnsZoneName  
  // DNS zones - even private - are global.  
  location: 'global'  
  tags: commonTags  
}  
  
var privateDnsVnetLinkGuid = guid(privateDnsZoneName, vnetId)  
  
resource privateDnsVnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {  
  name: 'dns-link-${privateDnsVnetLinkGuid}'  
  parent: privateDnsZone  
  location: 'global'  
  properties: {  
    virtualNetwork: {  
      id: vnetId  
    }  
    // auto registration of VMs is not needed  
    registrationEnabled: false  
  }  
  tags: commonTags  
}  
  
output privateDnsZoneId string = privateDnsZone.id  

The privateDnsZoneId output is the one used in the template I wrote above, so the vnet connection is already there.

Of interest (for someone stuck with the same issue), my current workaround the issue is to create the A record "manually" in the private DNS zone, that contains the private IP of the private endpoint's network interface. It works well - although it questions the need of the DNS zone group.

Answer 5

I can't see anything obviously wrong with Remi's code, but for those of you arriving here looking for an answer, there are several resources that need to be configured correctly for this to work:

1. Private Endpoint
2. Private DNS Zone
3. DNS Zone Group
4. VNet Link

If you would like to review a tested and working example (including Azure Redis Cache, Storage and Key Vault, take a look at: https://github.com/Azure-Samples/highly-available-zone-redundant-webapp

I have also compiled some troubleshooting notes here: https://github.com/DanielLarsenNZ/examples/blob/main/docs/pep-troubleshooter.md