Function App to write data to Storage Account with Private Link

Sumeetha Mogasati 126

Hi,
Function App has been successfully deployed that writes data to Storage Account using the Storage Account Key.

The function app is working as expected for the Storage Account with no Private Link.

When the same Function App (using the Storage Account Key) is deployed to a Storage Account that is configured to use Private Link, function app is failing with an exception 'This request is not authorized to perform this operation'. Status: 403 (This request is not authorized to perform this operation.).

Please note the exception stack below. Help appreciated.

The listener for function 'myfunctionapp' was unable to start. This request is not authorized to perform this operation.

RequestId:491a211d-601e-004a-1c07-63c526000000

Time:2022-05-08T18:15:44.6350205Z

Status: 403 (This request is not authorized to perform this operation.)

ErrorCode: AuthorizationFailure

Content:

<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.

RequestId:491a211d-601e-004a-1c07-63c526000000

Time:2022-05-08T18:15:44.6350205Z</Message></Error>

Headers:

Server: Microsoft-HTTPAPI/2.0

x-ms-request-id: 491a211d-601e-004a-1c07-63c526000000

x-ms-client-request-id: 23cc799b-59c0-4787-a89d-494ae0166de9

x-ms-error-code: AuthorizationFailure

Date: Sun, 08 May 2022 18:15:44 GMT

Content-Length: 246

Content-Type: application/xml

Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException:

Azure.RequestFailedException:

at Azure.Storage.Blobs.BlobRestClient+<AcquireLeaseAsync>d__36.MoveNext (Azure.Storage.Blobs, Version=12.9.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Azure.Storage.Blobs.Specialized.BlobLeaseClient+<AcquireInternal>d__26.MoveNext (Azure.Storage.Blobs, Version=12.9.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Azure.Storage.Blobs.Specialized.BlobLeaseClient+<AcquireAsync>d__25.MoveNext (Azure.Storage.Blobs, Version=12.9.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Microsoft.Azure.WebJobs.Script.BlobLeaseDistributedLockManager+<TryAcquireLeaseAsync>d__12.MoveNext (Microsoft.Azure.WebJobs.Script, Version=3.5.0.0, Culture=neutral, PublicKeyToken=null: D:\a_work\1\s\src\WebJobs.Script\Host\DistributedLockManagers\BlobLeaseDistributedLockManager.cs:134)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Microsoft.Azure.WebJobs.Script.BlobLeaseDistributedLockManager+<TryLockAsync>d__8.MoveNext (Microsoft.Azure.WebJobs.Script, Version=3.5.0.0, Culture=neutral, PublicKeyToken=null: D:\a_work\1\s\src\WebJobs.Script\Host\DistributedLockManagers\BlobLeaseDistributedLockManager.cs:82)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Microsoft.Azure.WebJobs.Host.SingletonManager+<TryLockAsync>d__19.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.32.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonManager.cs:113)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener+<StartAsync>d__13.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.32.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonListener.cs:48)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+<StartAsync>d__13.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.32.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:68)

Accepted answer

MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-10T09:35:04.333+00:00

@Sumeetha Mogasati ,

Thanks for reaching out to Q&A.

I hope the private endpoint linked with the Storage account and the function app are in the same vnet. They can be in same or different subnets inside the vnet. Also please make sure you have configured the DNS changes for private endpoints properly.

From the function app settings perspective, please make sure you have added the below settings,

https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings?msclkid=ccba4026d04311eca281f0c501e04a01#website_dns_server

I hope this helps!

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.
Please sign in to rate this answer.
Sumeetha Mogasati 126 Reputation points

2022-05-10T11:27:31.073+00:00

Thanks for your help, MughundhanRaveendran-MSFT.

So the Storage Account (SA) Key does not work to access the Function App when the SA is secured using the Private Link (Private EndPoint)?

Thanks

MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-10T13:10:38.127+00:00

@Sumeetha Mogasati , Function app will be able to access the storage account linked with private endpoint if it is configured as per the above answer

Sumeetha Mogasati 126 Reputation points

2022-05-12T09:03:47.623+00:00

Thanks for your prompt help and rapport.

Your above suggested approach is being implemented now. Can provide feedback after testing to seek any feedback or accept your answer ASAP.

Regards

Sumeetha Mogasati 126 Reputation points

2022-05-13T23:05:32.703+00:00

Hi MughundhanRaveendran-MSFT

The Function App was deployed using the 'Consumption Plan' and no Networking features are available to deploy to the same VNET as the Private EndPoint.

It appears that a new Function App must be created using the 'Premium plan' or 'App Service Plan'. Please suggest, which plan must be selected?

Regards,

MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-14T07:15:35.027+00:00

@Sumeetha Mogasati , Both premium and App service plan supports Vnet. If you would like to have a dedicated plan you can go for App service plan, if you would like to choose a serverless mechanism then you can choose Premium plan.

Sumeetha Mogasati 126 Reputation points

2022-05-14T10:11:16.333+00:00

Thanks for your help and rapport. Please note a couple of queries for your help/input.

Premium Plan selected for the Function App, while adding VNet Integration, it is required to enter a CIDR block as shown below. There are existing a few subnets with the address ranges.

Please inform/help with the CIDR block? What would be this CIDR block value?

Another issue is existing Azure Policy is preventing adding suggested below AppSettings. Are these AppSettings mandatory, or please inform an alternative?

MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-16T13:48:46.96+00:00

@Sumeetha Mogasati ,

You will have to enter the subnet address block, here is an example

Also both the app settings are required for Vnet integration
https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_dns_server
https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentovervnet

Sumeetha Mogasati 126 Reputation points

2022-05-16T21:24:33.977+00:00

Hi @MughundhanRaveendran-MSFT

Thanks for your persistent help and rapport. Please note the latest troubleshooting steps below. Network issues are resolved, just the permissions on the Storage Account for the Function App to fix.

Instead of creating the SubNet from the Function App --> Networking Configuration page, taken an alternative approach and created a SubNet from the existing VNet (same as the Storage Account Private endpoint).

Then a new Function App with the Premium Plan was deployed.

From the Function App-> Networking --> Outbound Traffic --> VNet integration, integrated the Function App to the previously created SubNet in the above Step 1. No CIDR to worry about anymore :-)

Also, I successfully added your suggested configuration to the App Settings. The Policy I mentioned in the previous post was irrelevant, which was resolved with no issue.

The Function App successfully accessed the Storage Account (HTTP 200) but failed to write/update the content with 403. Please note the below-copied screenshots where sensitive info is removed.

The Function App should write/update content on a named container and update data within the $web container on the Storage Account.
When it tries to write/update the content on the $web container, HTTP 403 (HEAD) is thrown, and the other named container is throwing HTTP 403 for the PUT operation.

Does it appear that the resolution may involve granting the Function App correct permissions on the Storage Account?
Further help is greatly appreciated.

Regards,

Sumeetha Mogasati 126 Reputation points

2022-05-16T21:25:42.813+00:00

Please note the screenshots below.

MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-17T04:53:18.417+00:00

@Sumeetha Mogasati , How are you writing files to the storage account? Are you using the output binding or using the Storage SDK? Would you be able to share the code?

Sumeetha Mogasati 126 Reputation points

2022-05-17T09:46:56.51+00:00

Hi @MughundhanRaveendran-MSFT

Maybe it is too early to say that the Function App successfully accessed the Storage Account.
The logs via Log Stream (copied below) and the 'End to end Transaction Log' from the App Insights indicate that the Function App is still suffering to access and write the content?

I am using the Storage SDK to write files, as the same source working/writing files to the other Storage Account, it may not add value to share the code? But happy to do so, please confirm.

2022-05-17T08:30:00.045 [Information] Executing 'UploadFiles' (Reason='Timer fired at 2022-05-17T08:30:00.0442901+00:00', Id=319858d6-4bf4-43d3-9dca-6fe5988ae6ec) 2022-05-17T08:30:00.048 [Information] C# Timer trigger function Started at: 5/17/2022 8:30:00 AM 2022-05-17T08:30:04.786 [Error] This request is not authorized to perform this operation.

Sumeetha Mogasati 126 Reputation points

2022-05-17T09:49:43.177+00:00

Please note the functionality of the Function App below. The Function App was successfully tested with a separate Storage Account with no Private Link (Private Endpoint), and all are working as expected. It has been challenging to get the App work with the Storage Account secured with a Private Link (Private Endpoint).

Function App executes at the scheduled intervals.

Download a .exe file and a regular file from the provided URL and upload it to the Storage Account named container.

Access the JSON file stored within the $web container on the Storage Account and update some data in it.

As described in the above response, a SubNet was created within the same VNet as the Storage Account Private Endpoint, and then the Function App was integrated successfully.
Maybe the NSG on the VNet is the root cause for the current error? As the SubNet is within the VNet, the traffic is allowed by default. Also, checking the NSG rules appears to be accurate. Please note the Function VNet integration with IP addresses and the VNet NSG Inbound below.

----------

VNet NSG Inbound Rules

MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-17T12:52:20.837+00:00

@Sumeetha Mogasati , How does the Storage account firewall configuration look like? I hope it is in the same vnet as Functions. Can you try to whitelist the ip in the Storage account networking settings ?

Sumeetha Mogasati 126 Reputation points

2022-05-17T13:32:15.04+00:00

Hi @MughundhanRaveendran-MSFT

Once again thank you for your persistent help and insight.

Yes, the Storage Account is within the same VNet as the Function App. Both (Storage Account & Function App) are in two separate SubNets within the same VNet.

Yes, IP can be whitelisted in the Storage Account Networking Settings.

The IP you refer above is it for the Function App ? that is accessible from the Function --> Properties --> Virtual IP address. Please inform?

Please note the Storage Account Firewall config below.

Sumeetha Mogasati 126 Reputation points

2022-05-17T13:41:50.987+00:00

Hi @MughundhanRaveendran-MSFT

I think that the Function App IP addresses can be obtained as documented at https://learn.microsoft.com/en-us/azure/azure-functions/ip-addresses?tabs=portal#find-outbound-ip-addresses

There are many Possible Outbound IP Addresses available. Does all of these needs to be whitelisted in the Storage Account Firewall?

Please let me know.

Regards,
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

2 additional answers

Takahito Iwasa 4,851 Reputation points MVP Volunteer Moderator

2022-05-09T21:39:07.77+00:00

Hi, @Sumeetha Mogasati

Is your function configured to access your storage account using a private link?

https://learn.microsoft.com/en-us/azure/azure-functions/functions-networking-options?tabs=azure-cli#private-endpoints

To call other services that have a private endpoint connection, such as storage or service bus, be sure to configure your app to make outbound calls to private endpoints.
Please sign in to rate this answer.
Sumeetha Mogasati 126 Reputation points

2022-05-10T09:09:19.33+00:00

Hi Takahitolwasa

Thanks for your response and help.

The function is configured to access storage account using the storage account key, not the Private Link.

Takahito Iwasa 4,851 Reputation points MVP Volunteer Moderator

2022-05-10T11:29:38.687+00:00

You can only connect to storage with private endpoints enabled from the associated VNET. So you need to connect the function to the storage via VNET using a private link.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Sumeetha Mogasati 126 Reputation points

2022-05-17T14:13:59.587+00:00

Thanks for your rapport and help, @MughundhanRaveendran-MSFT

can confirm that after adding the Function App SubNet to the Storage Account Firewall, all working as expected now.

Regards
Please sign in to rate this answer.
MughundhanRaveendran-MSFT 12,506 Reputation points

2022-05-18T13:08:49.58+00:00

@Sumeetha Mogasati , Thank for the update
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Your answer