question

hnnsj-1436 avatar image
0 Votes"
hnnsj-1436 asked hnnsj-1436 commented

Azure Function occasionally fails because of invalid access to socket

I'm running an Azure Function on a consumption plan that has been up and running in production for a couple of years. In the past two-three weeks the function occasionally (but rarely) fails with the following error message:

An attempt was made to access a socket in a way forbidden by its access permissions.

Some research led me to believe that the App Service Plan it belonged to was running out of available sockets, and I moved the app to its own App Service Plan that no other app belongs to. But after a couple of days the exception started occuring again. Since it's running on a consumption plan there's no way that I'm aware of to control the number of available sockets.

The function runs thousands of times per day but I only get this error perhaps once or twice per day. It has also been running fine for a couple of years until now. So I'm rather sure it's not some obvious misconfiguration.

azure-functions
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TakahitoIwasa avatar image
0 Votes"
TakahitoIwasa answered

Hi, @hnnsj-1436

Perhaps you will have problems when you have reached the outbound limit.
In the case of App Service Plan, there is an upper limit for each plan.

https://github.com/MicrosoftDocs/azure-docs/blob/main/includes/functions-limits.md
https://blog.vjirovsky.cz/azure-functions-and-forbidden-socket-exception/

As a fundamental measure, in the case of App Service Plan, it is necessary to scale up and consider a mechanism for reusing communication on the application side.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MughundhanRaveendran-MSFT avatar image
0 Votes"
MughundhanRaveendran-MSFT answered hnnsj-1436 commented

@hnnsj-1436 ,

Thanks for reaching out to Q&A.

Given the frequency of the issue, it appears to be transient issue which is common in cloud computing world. Having a retry mechanism should take care of transient issues.

However, if the issue is seen consistently/in a certain pattern, you will have to look into the outbound connections of the function app. The number of available connections in a Consumption plan is limited partly because a function app in this plan runs in a sandbox environment. One of the restrictions that the sandbox imposes on your code is a limit on the number of outbound connections, which is currently 600 active (1,200 total) connections per instance. When you reach this limit, the functions runtime writes the following message to the logs: Host thresholds exceeded: Connections. For more information, see the Functions service limits.

Please look into this article to manage connections in function app : https://docs.microsoft.com/en-us/azure/azure-functions/manage-connections?tabs=csharp

I hope this helps!

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@hnnsj-1436 ,

Following up to see if the above answer helps. Do let me know if you have any queries.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

0 Votes 0 ·
hnnsj-1436 avatar image hnnsj-1436 MughundhanRaveendran-MSFT ·

To which logs is the following message written: Host thresholds exceeded: Connections? I can't find it in the logs, even though I continue to get the "An attempt was made to access a socket in a way forbidden by its access permissions."

I already use static connection clients (I use a QueueClient for ServiceBus and two TableClient instances for TableStorage). I can't see that any connections are opened per execution of the function.

0 Votes 0 ·
hnnsj-1436 avatar image hnnsj-1436 MughundhanRaveendran-MSFT ·

I've managed to pin down the error to an EventHub output binding. I have the following binding for my function:

 [EventHub("%OutgoingEventHubName%", Connection = "OutgoingEventHubConnectionString")] IAsyncCollector<string> outputEvents

My function generates multiple output events per incoming event, which are then added using

 await outputEvents.AddAsync(serializedMessage);

It is this line that sometimes throws an exception:

 An attempt was made to access a socket in a way forbidden by its access permissions.
 at Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient.CreateConnectionAsync(TimeSpan timeout)
 [...]
 at Microsoft.Azure.WebJobs.Host.Bindings.TypedAsyncCollectorAdapter`3.AddAsync(TSrc item, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Bindings\AsyncCollector\TypedAsyncCollectorAdapter.cs:line 40
 [...]
0 Votes 0 ·