Best way to handle db connection in Azure Function handling that is robust.

Question

I'm looking for a robust way to to handle database failres in Azure Function in a robust manner. In this example provided to use Lazy to reuse connection.

https://learn.microsoft.com/en-us/azure/azure-functions/manage-connections?tabs=csharp

private static Lazy<CosmosClient> lazyClient = new Lazy<CosmosClient>(InitializeCosmosClient);  
private static CosmosClient cosmosClient => lazyClient.Value;  
  
private static CosmosClient InitializeCosmosClient()  
{  
    // Perform any initialization here  
    var uri = "https://youraccount.documents.azure.com:443";  
    var authKey = "authKey";  
     
    return new CosmosClient(uri, authKey);  
}  

However, the code above is not robust, if the database is not available, the Lazy caches the wrong value. This actually happeneed to a deployment, database was shutdown, an API was called, and error persists even when the database came back online. I had to manually restart the Azure Function from portal.

I tried the Lazy with LazyThreadSafetyMode.PublicationOnly but it didn't work. What's the correct way to store db connection

1. Robust to handle if database has connections errors, but comes back online later without needing to restart Azure Function.
2. In scalable way across multiple instances

Answer 1

@Robin Tan ,

Thanks for reaching out to Q&A.

You can explore Polly retry mechanism with .net core HttpClient : https://www.hanselman.com/blog/adding-resilience-and-transient-fault-handling-to-your-net-core-httpclient-with-polly

https://azure.microsoft.com/en-in/blog/using-the-retry-pattern-to-make-your-cloud-application-more-resilient/

https://github.com/TroyWitthoeft/AzureFunctionHttpClientFactoryPollyLogging

Answer 2

Hi.
When it comes to Best Practice of storing secrets in Azure, there are couple of ways:

1. Connections strings should never be stored in code. Use Application Settings. For local development of Azure Functions - you can look at local.settings.json file. You application settings are inside a node called values. You just create a key:value pair. In Production - the keys inside values node needs to be created in Application Settings section of the Azure FunctionApp resource. Then in your code - you can just reference the connection string key. So the code never stores the value but rather at run time gets it from the application settings.
2. Use key vault and create a secret. Create a managed identity for your azure function app. In key vault policies add a policy to allow secrets read for azure function managed identity. Then in your function, create a application setting with key = your connection string name and value = @Microsoft.KeyVault(VaultName=myvault;SecretName=mysecret). More info here: https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references

Above are from the perspective of storing the connection string. A robust way of making sure you connect to database is to use what @MughundhanRaveendran-MSFT suggested - Polly. Polly allows you to set up retry policy. You can say things like retry 3 times and then fail. Meaning if first time code was not able to connect due to database server issue - it will try second time with a delay of 5 seconds in between. If database server is back up in that 5 seconds - the second try will have a successful connection and your don't have an issue in your application.

Hope this helps.