How to secure access to host storage account for Function Apps on Windows Elastic Premium Plan with VNET Integration

Question

How to secure access to host storage account for Function Apps on Windows Elastic Premium Plan with VNET Integration

ZQadir 195

Background / Setup:

Function Apps on Windows Elastic Premium Plan (EP1) with Zone Redundancy and Auto Scaling from 3 to x nodes.
Inbound access control on main site and advanced tool site of the Function App. Deny all for advanced tool site with temporary whitelisting of deployment agent IP for any new deployments.
VNET Integration for outbound access control. Respective service endpoints enabled on the subnet used for VNET Integration. Target resources (business data storage accounts, key vault, SQL servers, service bus etc.) have Network ACLS to allow access for the subnet used for VNET integration by Function Apps. Target resources do have IAM/RBAC using the System Assigned Managed Identity of relevant Function Apps. Access to such target resource is working with Service Endpoints and Managed Identity using the latest extensions in Function Apps.
Functions are coded in C# using .NET 6, and deployed using Azure DevOps Pipelines.
Function Apps have FUNCTIONS_EXTENSION_VERSION = ~4, FUNCTIONS_WORKER_RUNTIME = dotnet, WEBSITE_CONTENTOVERVNET = 1 and vnetRouteAllEnabled
Separate host storage accounts for each of the function apps, configured as Standard, General Purpose V2 with Zone Redundant Storage

Question:

How can we best secure access to the host storage account used by each of the Function App on Windows Elastic Premium Plan that uses VNET Integration?

Our preferred solution for this will be:

IAM/RBAC on the host storage account using system assigned managed identity of the function app, so that we don’t have to worry about rotating storage account keys.
Implement Network ACLs on host storage account so that only the subnet that’s used for VNET integration by Windows Elastic Premium Plan and Function apps are allowed access to it, using the Service Endpoints.

Hurdles in our preferred solution:

We have tried implementing managed identity based access for host storage account instead of the typical storage account key based approach. However, the managed identity based access isn’t supported for the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING setting, which is needed for Windows Elastic Premium Plan. So, we have to use storage account key based access for this setting. Moreover, managed identity based access is still in preview. So, we parked that and for now and ended up in using Account Key based access to host storage accounts of Function Apps for both AzureWebJobsStorage and CONTENTAZUREFILECONNECTIONSTRING. References: https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference?tabs=blob#connecting-to-host-storage-with-an-identity-preview and How to use managed identity for WEBSITE_CONTENTAZUREFILECONNECTIONSTRING · Issue #86604 · MicrosoftDocs/azure-docs · GitHub
We have tried implementing the Network ACLs on the host storage account; by provisioning the storage account with Network ACLs upfront, and then using storage account when provisioning the Function Apps that has VNET integration to the subnet that’s whitelisted on the storage account, and has service endpoints enabled. We made the function app to functionally work. However, we noticed a few Failures reported in the dashboard when we go to the Insights of that host storage account in Azure Portal. Expanding those failures, show errors related to File service with various error codes. Looking at the Microsoft documentation for implementing Network ACLs on host storage account with service endpoints based access for Function Apps here, https://learn.microsoft.com/en-us/azure/azure-functions/configure-networking-how-to#restrict-your-storage-account-to-a-virtual-network, we noticed that it follows a two step process i.e., provision a function app with a storage account that has not got the Network ACLs and no Service Endpoints based access, and later replace it with another storage account that has Network ACLs and Service Endpoints based access. Not sure what’s the reason for that two step approach as compared to the one step approach that we follow, and if the errors on the host storage account are due to this deviation from Microsoft outlined 2 step approach.

Concluding remark:

Keeping the above setup in view, what’s the best officially supported way to secure access to the host storage accounts for the Function Apps (even if that way is different than our preferred solution) running on Window Elastic Premium Plan with VNET Integration?

ZQadir 195 Reputation points

2023-01-24T14:39:08.7433333+00:00

Comment posted twice.

Answer accepted by question author

0 additional answers

Your answer

ZQadir 195 Reputation points

2023-01-24T14:39:08.7433333+00:00

Comment posted twice.

Answer 1

MughundhanRaveendran-MSFT 12,511

@ZQadir

Thanks for the detailed explanation and insights.

"IAM/RBAC on the host storage account using system assigned managed identity of the function app, so that we don’t have to worry about rotating storage account keys".

This would be the ideal approach however as you mentioned, the feature is still in preview and you will have to wait for it to become GA. Until that time, you can continue to use the storage account keys

"Implement Network ACLs on host storage account so that only the subnet that’s used for VNET integration by Windows Elastic Premium Plan and Function apps are allowed access to it, using the Service Endpoints".

This is also the recommended approach, there is no need to create two storage account. The first storage account that was created can be used to implement ACL. Could you please let me know what is the error that you see in the File service? Please provide screenshot.

ZQadir 195 Reputation points

2023-02-04T11:06:51.6466667+00:00
Hi @MughundhanRaveendran-MSFT

Many thanks for the response.

Regarding the support for system assigned managed identity for host storage account setting WEBSITE_CONTENTAZUREFILECONNECTIONSTRING, the Github link that I shared is just over a year old. So, was wondering if this feature is on the backlog for product team and when to expect GA for this?

For the Network ACLs and Service Point Based Access to Host Storage Accounts, many thanks for the confirmation that the single step setup of Host Storage Account provisioned with Network ACL for access via Service Endpoints is OK too instead of a two step process which involves replacing the storage account without Network ACL to one with Network ACL afterwards.

With regards to the errors reported on Host Storage Account (that has Network ACLs and Service Endpoints based access) for Function app, we have the full diagnostic settings (that includes reads, writes, deletes and metrics) enabled for that storage account with logs routed to Log Analytics Workspace (screenshot below)

So, when we look at the Insights of that Storage Account under monitoring and look at the summary of failures in last 7 days, this is what we get:

Please note that this is a POC function app with almost no traffic (a couple of hits in a day at max), and almost all these errors do come when we do the deployment or change any configuration value, even though the function app functionally works. Here is further breakdown of those from the Failures Screen (once scrolled down)

When we click on Show Logs to see the details and run the last run query in Log Analytics Workspace, we see the query configured for the Failures Dashboard as below:

let serviceValues = dynamic(['*']); let operationValues = dynamic(['*']); let statusValues = dynamic(['*']); StorageBlobLogs | union StorageQueueLogs | union StorageTableLogs | union StorageFileLogs | where StatusText != "Success" | where "*" in ('*') or ServiceType in ('*') | where "*" in ('*') or OperationName in ('*') | where "*" in ('*') or StatusText in ('*') | extend Service = ServiceType | extend AuthType = AuthenticationType | extend CallerIpAddress = split(CallerIpAddress, ":")[0] | summarize ErrorCount = count() by Service, OperationName, StatusText, StatusCode, AuthType, tostring(CallerIpAddress), Uri | sort by ErrorCount desc

Here are the quick stats of the various error codes by operations and count for the last 7 days:

One thing that we noticed is that Failure Dashboard query is looking for StatusText != "Success", which is good way to filter any failures for Blob Service which relies on HTTP error codes and status text. However for the file service using SMB protocol, it should have excluded the Status Code 0 too (that's we have coloured them in green ). We are specifically concerned about the -ive status code for the file service and the volume of those for a POC app without any traffic. On reading a bit about file service and SMB protocol used, we found that the status code for that are in hex reference: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-cifs/8f11e0f3-d545-46cc-97e6-f00569e3e1bc. So, on converting these -ive Status codes from decimal to Hex, we can see that some of the Hex values represent an error condition too. Here is the quick lookup that we created from these Status Codes in decimal to hex values and their descriptions:

What we are not sure is if we do need to worry about these errors from file service (as they can be a symptom to something going wrong, even though the function apps are functionally working in the test environments) OR these errors are bound to happen for host storage account and we must safely ignore these errors on Host Storage account for Azure Function Apps?
ZQadir 195 Reputation points

2023-02-04T11:07:24.1866667+00:00

@MughundhanRaveendran-MSFT

Many thanks for the response and the guidance that we can safely ignore those errors on the host storage accounts of function apps, as we are not getting the "Azure functions runtime unreachable" error in the Function App; and the functions are working as expected from the initial tests that we have done so far.
MughundhanRaveendran-MSFT 12,511 Reputation points

2023-02-04T11:07:40.42+00:00

@ZQadir , Sorry I dont have an ETA for identity for host storage account setting WEBSITE_CONTENTAZUREFILECONNECTIONSTRING to go GA. I will mention here once I have some updates.

These errors that you see in the storage account are safe to ignore as long as you dont see "Azure functions runtime unreachable" error in the Function app.

Share via

How to secure access to host storage account for Function Apps on Windows Elastic Premium Plan with VNET Integration

0 additional answers

Your answer