APIM Private Endpoint missing DNS CNAME alias

SvenGlöckner 441

We created an APIM stv2 instance, let's say test-apim.azure-api.net. We also created a private endpoint in our spoke vnet and a private DNS zone in the hub vnet. Both Vnets are peered. When doing a nslookup there is no test-apim.privatelink.azure-api.net entry. We already reprovisioned and created a new private endpoint serveral times but still no luck. What can be wrong?

KapilAnanth-MSFT 44,311 Reputation points Microsoft Employee

2023-04-27T06:28:28.1666667+00:00
@SvenGlöckner

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

The Private DNS Zone should be linked to the

VNet which contains a custom DNS server in case you use a custom DNS server.

All the VNets (both Hub and Spoke) in case you are using the default Azure Provided DNS

Now,

Do you have a custom DNS server or are you using the default Azure DNS

From where is the resolution failing? Is this from a VM in Hub or from a VM in Spoke?

If you feel that the entry is missing, you can directly check the DNS Zone "privatelink.azure-api.net" and manually add the A record "test-apim" pointing to the PE's IP

Cheers,

Kapil
SvenGlöckner 441 Reputation points

2023-04-27T06:38:20.8066667+00:00

Hi @KapilAnanth-MSFT

thanks for you comment. However, all of your questions or findings were already cleared or solved in the comment section of RevelinoB's answer. Also, please see my comment from today.

Once again: The Azure public CNAME entry test-apim.privatelink.azure-api.net is missing after creation of private endpoint.

From my understanding, the Azure public DNS adds the entry test-apim.privatelink.azure-api.net

after creation of private endpoint. When checking on my local machine with nslookup or using a tool like www.digwebinterface.com I can see that this entry is missing.
ChaitanyaNaykodi-MSFT 25,841 Reputation points Microsoft Employee

2023-04-27T16:06:37.23+00:00

@SvenGlöckner

Apologies for your experience here. I went through your comments below and if I understand correctly when you perform a nslookup for apimservicename.azure-api.net you are not able to view the apimservicename.privatelink.azure-api.net alias as it is not getting registered in your public DNS after private endpoint is enabled.

Based on my understanding above I am following a similar issue internally and the product team is currently working on a fix, and it will be rolled out soon. I will keep you posted on the progress.

Thank you!
SvenGlöckner 441 Reputation points

2023-04-27T18:16:52.2333333+00:00

@ChaitanyaNaykodi-MSFT

No need to worry or apologize. You fully understand my issue now. The public DNS entry for the privatelink is missing. My assumption is that the issue occured in the last 2 months.

Great that you're already working on it internally. Hope to get this fix soon.

Thank you! I'm looking forward hearing from you.
ChaitanyaNaykodi-MSFT 25,841 Reputation points Microsoft Employee

2023-05-03T15:51:58.14+00:00

@SvenGlöckner

As the product team is working on this fix, alternatively they are fixing this issue for the customers by updating the alias records from the backend directly.

If you need help in this regard, can you please send an email to azcommunity@microsoft.com with following details.

Subject: Attn Chaitanya

APIM endpoint name:

Subscription ID:

I will send an update here as soon as the fix is rolled out in production. Thank you!

Accepted answer

ChaitanyaNaykodi-MSFT 25,841 Reputation points Microsoft Employee

2023-05-05T16:59:03.87+00:00

@SvenGlöckner

Thank you for reaching out to us via email, I have just replied to your email.

Another update from on this issue is :
We've rolled out a fix for this issue and the fix is already in the prod and all newly created private endpoint for Azure API Management will no longer have the issue.

At the same time, we are working on fixing the existing broken private point cnames.

It will be helpful if you test this out and let us know if you are still facing this issue. Thank you!

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

2 additional answers

RevelinoB 2,780 Reputation points

2023-04-26T09:36:54.8133333+00:00

Hi Sven, If you're not seeing the test-apim.privatelink.azure-api.net entry in your nslookup results, there could be a few things causing the issue. First, make sure that your APIM instance is set up correctly to use private endpoints. Then, check that your private DNS zone is configured to resolve the private endpoint for your APIM instance. You should also verify that your VNet peering is set up correctly between your hub and spoke VNets. What you also could double check is if your DNS resolution settings to ensure that your VMs are using the right DNS server(s) to resolve private DNS names. I hope this could help with your issue?
Please sign in to rate this answer.
SvenGlöckner 441 Reputation points

2023-04-26T10:38:04.5133333+00:00

Hi RevelinoB, thanks for you answer. However, we already checked all these settings several times. Our private DNS zone is linked to the hub Vnet. Does it also need to be linked to the spoke vnet where the private endpoint reside?

RevelinoB 2,780 Reputation points

2023-04-26T10:48:25.4933333+00:00

Hi Sven, No, the private DNS zone does not need to be linked to the spoke VNet where the private endpoint resides. The private DNS zone should be linked only to the hub VNet where the APIM instance is connected. Based on what you described, it seems that you have already set up the private endpoint, private DNS zone, and virtual network peering correctly. However, the private DNS zone is not resolving the APIM instance's hostname. You can try a couple of things to cross-check and adjust if needed:

Check if that the private endpoint is correctly configured: Double-check that the private endpoint is correctly configured with the correct resource ID for the APIM instance, and that the private DNS zone is correctly linked to the hub VNet.

Double check the DNS resolution settings on the VNet: Verify that the virtual network where the APIM instance is located is configured to use the private DNS zone. You can check this by going to the "DNS servers" section of the VNet configuration in the Azure portal.

Check the DNS suffix search list on the client: If you are testing DNS resolution from a client machine, verify that the DNS suffix search list includes the private DNS zone name. This will ensure that the client machine can properly resolve the APIM instance's hostname.

Check if the APIM instance's hostname is registered in the private DNS zone: Ensure that the APIM instance's hostname is registered in the private DNS zone. You can verify this by checking the DNS records in the private DNS zone in the Azure portal. I hope this help? if you have any queries please let me know.

SvenGlöckner 441 Reputation points

2023-04-26T10:58:26.2666667+00:00

Hi RevelinoB, we checked all the settings. From the docs, Azure public DNS provides the privatelink alias after the creation of the private endpoint. This is something we cannot control. This privatelink alias is still missing when issuing nslookup, as I've already mentioned.

RevelinoB 2,780 Reputation points

2023-04-26T11:05:16.87+00:00

Hi Sven, You that is correct, the privatelink alias that is provided by Azure public DNS after the private endpoint is created, and that it is not something you can control directly. If the privatelink alias is still missing when you perform an nslookup, there could be a few possible explanations. What I would do is check or do the following

Verify that the private endpoint is in a "ready" state: Ensure that the private endpoint is in a "ready" state and has successfully completed the connection process with the APIM instance. You can check the status of the private endpoint in the Azure portal or by using the Azure CLI or PowerShell.

Check the NSG settings: Ensure that there are no network security groups blocking traffic between the APIM instance and the private endpoint. Also, verify that the necessary ports are open for traffic between the two resources.

Wait for DNS propagation: As I mentioned earlier, DNS changes can take some time to propagate across Azure. It's possible that the privatelink alias simply hasn't propagated yet. Wait a few minutes and try again. I hope these pointers will help you with the issue?

SvenGlöckner 441 Reputation points

2023-04-26T11:15:41.23+00:00

Hi RevelinoB, I'll try to answer you questions.

The private endpoint is in connection status "Approved" and Provisioning state "Succeeded".

We have no NSG configured. Other private endpoints, i.e. for app services are working correctly.

We're already waiting for several days now. Nothing changes. I reprovisioned the whole APIM instance by using IaC terraform provider, but still no luck. The public DNS alias is still missing.

RevelinoB 2,780 Reputation points

2023-04-26T11:27:59.3166667+00:00

Hi Sven, Based on the info provided, it seems that the private endpoint for the APIM instance is properly configured and connected, and that the issue may be specific to the APIM instance or Azure's DNS service. You can troubleshoot the issue further: you can try checking the DNS resolution logs in the hub VNet's DNS server, verifying the private DNS zone configuration and A record creation, checking for any NSG rules that may be blocking DNS traffic, and ensuring that the APIM instance is properly configured with virtual network integration and private DNS integration. If these steps do not resolve the issue, I would recommended you to contact Azure support for assistance, as they can help investigate the issue further and escalate it to the appropriate team if necessary. I hope this helped?

elabrowm-7745 5 Reputation points

2023-04-26T21:20:59.8866667+00:00

Hi
I had the same issue .

Do you have an update to share on what's going wrong ?

I create an apim and an private endpoint

I have a VM deployed on the same vnet that the private endpoint , everything is configured as described on documentation

Here is the results of my test :

nslookup -> test-apim.privatelink.azure-api.net : return private IP

nslookup -> test-apim.azure-api.net : return public IP

Thanks

SvenGlöckner 441 Reputation points

2023-04-27T06:07:16.9933333+00:00

Yesterday, we provisioned a new private endpoint. During creation of endpoint, we were seeing the correct CNAME entry test-apim.privatelink.azure-api.net in the public DNS by issuing nslookup for a short amount of time (only about 20 seconds). After that, it was gone and never came back.

So we're thinking this must be an issue on Azure side and we'll going to raise an Azure support ticket through our CSP today.

elabrowm-7745 5 Reputation points

2023-04-27T08:39:31.1533333+00:00

Thanks , Will do the same

elabrowm-7745 5 Reputation points

2023-04-27T12:58:04.6966667+00:00

Hi Sven,

Just got this answer from the support :

Regarding APIM, the reason you need to configure DNS resolution to use the private DNS zone associated with the private endpoint is because APIM uses a custom DNS domain name that is not automatically registered with the private DNS zone. By default, APIM instances use the DNS domain name "azure-api.net", which is a public domain name and not registered with the private DNS zone.

Therefore, to ensure that DNS resolution uses the private IP address of the private endpoint instead of the public IP address, you need to configure DNS resolution to use the private DNS zone and specify the private IP address of the private endpoint as the DNS server.

We apologize for any confusion that our previous email may have caused. We hope this explanation clarifies the reason behind the configuration for APIM and why it differs from the behavior of other Azure services such as storage accounts.

However, you can use the private IP address of the private endpoint instead of the FQDN. This will ensure that your traffic stays within the VNet and does not traverse the public internet.

APIM works differently from other services .
Hope it helps

SvenGlöckner 441 Reputation points

2023-04-27T13:09:31.4366667+00:00

Thank you for your update. However, I'm not really satisfied with this answer. We had private IP resolution with test-apim.privatelink.azure-api.net running correctly in the last 2 months. That means that the public CNAME entry had been applied successfully. Only a change in our configuration made the public CNAME entry disappear. The behavior with public CNAME entry has been the same for our App services (privatelink.azurewebsites.net) and databases (privatelink.database.windows.net).

Please keep in mind that private endpoint feature for APIM is generally available since March 2023. Before, it only was in the preview stage. My assumption is that something internally in Azure is wrong since then.

@elabrowm-7745
But perhaps your scenario is different then ours. We already have a private DNS zone for privatelink-azure-api.net with an A record for the private IP address.

elabrowm-7745 5 Reputation points

2023-04-27T13:53:52.0333333+00:00

Yes I think my scenario is a little bit different , I recently create the DNS Zone for privatelink-azure-api.net

SvenGlöckner 441 Reputation points

2023-04-27T13:57:38.29+00:00

After you've created private endpoint of your APIM instance, please do a nslookup test-apim.azure-api.net from your PC in the public internet and let me know if you see a hostname alias like test-apim.privatelink.azure-api.net

elabrowm-7745 5 Reputation points

2023-04-27T14:34:20.4866667+00:00

No I don't have a hostname with this alias

SvenGlöckner 441 Reputation points

2023-04-27T14:35:59.2166667+00:00

That's exactly the issue that I have. Please contact Microsoft support with this detail. This public CNAME entry (alias) is needed that private endpoint correctly resolves private IP addresses.

elabrowm-7745 5 Reputation points

2023-04-27T15:00:35.77+00:00

okay will update my case with that

Adrian Fletcher 15 Reputation points

2023-04-27T21:42:44.1466667+00:00

Hi Sven

More of an FYI - We are experiencing the same thing, raised it with MS and they have asked us to drop and recreate the Private endpoint.

We are hoping it will stick and not drop off as it did with you.

It is not an ideal fix if feel. Let you know if it fails and get more information.

SvenGlöckner 441 Reputation points

2023-04-28T05:28:00.31+00:00

Hi @Adrian Fletcher

thanks for you information. We already dropped and recreated the private endpoint several times. The public CNAME alias does show up for a short time - approx. about 20 seconds - and then disappeared. It never stayed there permanently.

Adrian Fletcher 15 Reputation points

2023-04-30T00:54:13.3866667+00:00

Hi Sven

No good news here, ours did not hold either, tossed it back at MS, and as to date we have heard nothing back from them.

Adrian Fletcher 15 Reputation points

2023-05-02T00:21:15.8133333+00:00

Update from me. MS have come back to saying
"Yesterday, our production team had fixed this issue for you."

I have checked and it appears to be recorrectly resolving to the private end point IP address. I will get our dev team to triple check their apps are now working.

How or what they fixed or caused it to stop working, I have asked the question and will be interested in what it was.

Heres hoping they have fixed it for you.

SvenGlöckner 441 Reputation points

2023-05-02T07:58:18.38+00:00

Hi Adrian, thanks for keeping us updated.

Currently, I've redeployed the private endpoint. The behavior is still the same - the privatelink CNAME alias appears for several seconds and then magically disappears again. Perhaps they'll have to rollout the fix for each Azure region independently.

Adrian Fletcher 15 Reputation points

2023-05-05T03:58:37.38+00:00

Hi

The fix is still working.

Here is the fix they did
"The issue is the private link CNAME missing, and our production team has manually updated the CNAME for this DNS record to fix this issue."

This fix is very easy to roll out per region ...... but not really an ideal Fix in my opionion.

I will try and push for why did it drop in the first instnace. Will not hold my breath though.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
SvenGlöckner 441 Reputation points

2023-09-22T07:06:06.9433333+00:00

@ChaitanyaNaykodi-MSFT

Hi again,

we again have issues with private endpoint of our APIM instance.

It seems that the Azure DNS does sometimes not resolve the private IP address from your private endpoint for APIM correctly. Then, the public IP address from our APIM instance is returned.

This happens several days for about 5-10 minutes.

Can you please help?

We'll contact Azure support in parallel.

Regards

Sven
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

APIM Private Endpoint missing DNS CNAME alias

2 additional answers

Your answer