Intermittent Azure Function Connectivity Issue with Restricted Key Vault Using Managed Identity

Richard Haigh 45

I am experiencing intermittent connectivity issues between an Azure Function and Azure Key Vault. My setup is as follows:

Azure Function with Virtual Network (VNet) integration, set to route traffic through a specific subnet. It is set to use a linux runtime and python stack. Confirmed with local deploys and test runs that requirements and runtime are not the cause. Requirements file is also correctly building dependencies.

The Function is configured to retrieve storage vault connection keys from Azure Key Vault using dynamic key URIs in the format @Microsoft.Keyvault(SecretUri=https://<key-vault-name>.vault.azure.net/secrets/<secret-name>/). These are passed in the application settings of the configuration.

The Key Vault has a service endpoint that restricts access to the subnet used by the Azure Function and trusted Microsoft services.

The Azure Function is assigned a Managed Identity with Key Vault Secret User role.

Function has been running successfully without any virtual network changes for approximately one year.

Despite this setup, which aligns with Microsoft's documentation and best practices confirmed by colleagues, the Azure Function randomly loses access to the Key Vault. The issue is temporarily resolved by removing and re-adding the Key Vault URI in the application settings, followed by restarting the app.

I am trying to understand:

Is it normal for Azure Functions to experience such blips in connectivity after continuous operation? Is the "turn it off and on again" approach a legitimate long-term solution that is sometimes required?
Have there been recent updates to the treatment of managed identities or trusted service traffic that could account for this?
If no to 1 and 2, is it more likely an unseen cascade effect is what resolved this?

JamesTran-MSFT 36,596 Reputation points Microsoft Employee

2023-11-08T23:36:39.5533333+00:00

@Richard Haigh

I wanted to check in and see if you had any other questions or if you were able to resolve this issue?

If you have any other questions, please let me know. Thank you for your time and patience throughout this issue.

If the information helped address your question, please Accept the answer. This will help us and also improve searchability for others in the community who might be researching similar information.
Richard Haigh 45 Reputation points

2023-11-09T13:32:34.85+00:00

Hi James

Thank you for joining the conversation. Unfortunately I was not able to get permission to open a service ticket so I am unable to delve in depth with Microsoft support.

However I did notice this is an on-going high priority issue under the library that is showing as un-resolved: https://github.com/Azure/Azure-Functions/issues/2023

Adding custom domain as a whitelisted IP seems to be the only resolution but this contradicts the entire point if VNet integration and funnelling all traffic via the virtual network.

1 answer

MayankBargali-MSFT 70,136 Reputation points

2023-11-01T11:26:22.7466667+00:00

@Richard Haigh Thanks for reaching out.

Based on the information that you have provided it is not normal for Azure Functions to experience such blips in connectivity after continuous operation until and unless any configuration changed at the key vault, network or function end and there could be several reasons why this is happening.

Regarding your question about the "turn it off and on again" approach, while it may temporarily resolve the issue, it is not a legitimate long-term solution. It is important to identify the root cause of the issue and address it accordingly.

For your second point this could confirm post reviewing the logs. It is difficult to determine the exact cause of the issue without further investigation. I recommend checking the network and firewall settings, validate operation logs of your function app, key vault if there was any update operation was performed and in case if you are not able to find the root cause then this would need one on one support to review the logs if we can find the root cause. I suggest you to create a support ticket to assist you further. In case if you don't have the support plan then please let me know.
Please sign in to rate this answer.
Richard Haigh 45 Reputation points

2023-11-01T14:35:08.1866667+00:00

Hi @MayankBargali-MSFT

Thank you for your response and suggestions. I've thoroughly investigated the issue with Azure Functions experiencing intermittent connectivity problems with Azure Key Vault and have confirmed it to be a networking-related issue. Below is a detailed account of the steps I've taken to diagnose and confirm the nature of the problem:

VNet Integration: I verified the Virtual Network (VNet) integration settings for the Azure Functions to ensure there were no misconfigurations that could lead to connectivity issues. The VNet integration is configured correctly, and no changes were made to this setup during the period when the connectivity issues were observed. The subnet the function is integrated with is designated to Microsoft.ServerFarms and the correct service endpoints for Key Vault are enabled. There are no additional firewall rules.

Minimum TLS Version: I checked the Key Vault's minimum TLS version setting to ensure it wasn't causing a handshake failure with the Azure Functions. The TLS version is using minimum 2.1 compatible with Azure Functions, and no changes were made to this setting recently.

DNS Server Resolution: I tested DNS resolution within the VNet to ensure that the Azure Functions could resolve the Key Vault's address by creating an Ubuntu VM. DNS resolution was functioning correctly, and there were no issues with the DNS servers.

RBAC Identity Access: I reviewed the Role-Based Access Control (RBAC) settings to confirm that the Azure Functions' identity had the appropriate access to the Key Vault. The access permissions were in place and unchanged during the time of the connectivity issues. I tested granting the same RBAC permissions to a VM I created within the same VNet and it was able to retrieve the secret keys using the AZ CLI.

Custom Domain Whitelisting: As a crucial step, I added the custom domain name of the Azure Functions to the Key Vault's whitelist. This action effectively bypassed the intermittent connectivity problem, confirming that it is a network-related issue.

Network and Firewall Settings: I thoroughly reviewed the network and firewall settings for any inadvertent changes that might have caused the issue. No recent changes were found that could explain the connectivity problems.

Operational Logs Review: I examined the operation logs of both the Azure Functions and the Key Vault for any clues. No update operations or other anomalies were identified during the times when connectivity issues were observed. The function is passing the key vault URI in its application settings and so simply hits an internal server error. Logging statements show that is is simply importing the key vault uri in the correct format as al iteral "@icrosoft.keyvault" string, causing the crash when the function tries to use these access keys.

These steps confirm that the issue lies within the network settings, though the exact cause remains elusive. The temporary workaround of whitelisting the custom domain name in the Key Vault's firewall effectively resolves the connectivity problem but isn't a viable long-term solution.

Given the extensive diagnostics I've performed, I believe this situation warrants a more in-depth investigation that may require one-on-one support. I currently do not have a support plan in place. If you could assist in facilitating the creation of a support ticket or guide me on the next steps without a support plan, I would greatly appreciate it.

Thank you for your time and assistance in this matter.

MayankBargali-MSFT 70,136 Reputation points

2023-11-02T14:50:06.5+00:00

@Richard Haigh Please send an email with subject line “Attn:Mayank” to AzCommunity[at]Microsoft[dot]com along with your subscription ID so I can enable one time support request on your subscription.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Intermittent Azure Function Connectivity Issue with Restricted Key Vault Using Managed Identity

1 answer

Your answer