Share via

Azure Function Error: The operation has timed out

Michael Gant 21 Reputation points
2020-11-24T08:56:21.677+00:00

Hi,

An error has started popping up in a my Azure Data Factory Pipeline. I have a few Azure Function steps in the pipeline, but for some reason, one of the Azure Function steps has started returning an error. In Azure Data Factory, the error is a 3608 code after running for 1 minute 40 seconds:

Failure type: User configuration issue
Details: Call to provided Azure function 'CollateSheetsHTTPTrigger' failed with status-'InternalServerError' and message - 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'.

However, in a prior run sub-pipeline, this Azure Function ran successfully on the same data (parameters and worksheet are on the only difference). The subsequent 3 runs of pipelines fail immediately (after 2 seconds) at the first Azure Function (a different AZ function now) step in each, with the same 3608 error code but different details:

Call to provided Azure function '???????????????' failed with status-'NotFound' and message - '<html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> '.

Now it gets even stranger. After these 3 failed pipelines, the next pipeline which is pretty much the same as the previous 4 except for a few parameters, runs successfully, even though it has the same 2 AZ functions that failed before. And then the next 2 pretty similar pipelines also run successfully.

I then went and looked at the monitoring page for the 2 Azure Functions:
The first AZ function that failed, had 2 errors even though it only failed once in AZ Data Factory... the timing is slightly different for the 2 errors but they could only come from the first failed pipeline, so why does it say there are 2 errors? Then if you look at the actual error, all it says is "The operation timed out". The function was not running for more than 150 seconds so this is strange. Additionally, I have a bunch of error catching code and nothing comes up there.

The other failed AZ function steps from the other function do not show up on the monitoring page, it seems as if the first error crashed the AZ function app and then it eventually restarted?

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,016 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,728 questions
Accepted answer
  1. MayankBargali-MSFT 70,461 Reputation points
    2020-11-26T11:17:11.377+00:00

    Hi @Michael Gant

    Sharing our offline analysis.

    In the previous version of Function App, it was introduced a mechanism to ensure functions with prolonged Load Requests would timeout. This started to cause a regression where customers using Consumption and Premium functions in Linux would experience intermittent timeouts.

    This is supposed to be fixed by the latest version, which in your case was made available today. At 2020-11-26 02:58 it was already available as your functions were already using it. Due to this, we must observe if the issue will happen again.

    For Python please note that this may mean they run into an issue where the Functions are timing out because of debugging messages overwhelming the gRPC channel due to “extensive logging”.
    Pass log-levels for Functions to the Language Worker · Issue #6879 · Azure/azure-functions-host (github.com)

    You will monitor it for some more time and will let us know if you are still observing the issue.

    Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

    1 person found this answer helpful.
    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.