ADF calling Azure Function: '404 not found' exception once out of every 10 runs (after 2 seconds)

Question

ADF calling Azure Function: '404 not found' exception once out of every 10 runs (after 2 seconds)

Connor Hazen 1

I am, seemingly at random, receiving 404 not found errors on azure function calls from within ADF pipeline. They occur once every 10 runs, roughly, and come in two forms:

The first has this HTML as its error:

Call to provided Azure function 'Parse' failed with status-'NotFound' and message - '<html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> '.

The second is:

Call to provided Azure function 'Parse' failed with status-'NotFound' and message - 'Invoking Azure function failed with HttpStatusCode - NotFound.'.

These are interspersed within successful runs and rerunning the pipeline with the same triggered file (this a blob upload trigger) will run with no problems suggesting that it is not an input issue. Furthermore, the fact that they occur 2 seconds means it is not a timeout issue? Any thoughts?

MartinJaffer-MSFT 26,236 Reputation points

2021-05-25T22:49:11.767+00:00

Hello @Connor Hazen and welcome to Microsoft Q&A.

I did a little digging and found a similar case from around a year ago. In that case the cause was an overly long URL / query string (>2000 characters).

Could you give me an idea of how long the URL / query string is, and how frequently you are calling the function?

Azure function is not in my area of expertise, so I also wanted to consider possibility like overwhelming the endpoint.
Connor Hazen 1 Reputation point

2021-05-26T01:28:15.287+00:00

Hi @MartinJaffer-MSFT thank you for helping me debug this issue.

We are running the function currently 20-50 times a day. However that is going to scale up in the near future to a 1000+ a day.

As far as the URL string, it is simply the function app url, key, and three body params that are between 10 to 100 characters. There are no headers. So it is not an overlong URL string.

Again, thank you.
MartinJaffer-MSFT 26,236 Reputation points

2021-05-26T22:23:33.78+00:00

Thank you for letting me know length isn't the problem, @Connor Hazen .

I would like to pass your case into the care of my colleague @ChaitanyaNaykodi-MSFT . Azure Functions falls under his area of expertise.

5 answers

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2021-05-25T22:49:11.767+00:00

Hello @Connor Hazen and welcome to Microsoft Q&A.

I did a little digging and found a similar case from around a year ago. In that case the cause was an overly long URL / query string (>2000 characters).

Could you give me an idea of how long the URL / query string is, and how frequently you are calling the function?

Azure function is not in my area of expertise, so I also wanted to consider possibility like overwhelming the endpoint.
Connor Hazen 1 Reputation point

2021-05-26T01:28:15.287+00:00

Hi @MartinJaffer-MSFT thank you for helping me debug this issue.

We are running the function currently 20-50 times a day. However that is going to scale up in the near future to a 1000+ a day.

As far as the URL string, it is simply the function app url, key, and three body params that are between 10 to 100 characters. There are no headers. So it is not an overlong URL string.

Again, thank you.
MartinJaffer-MSFT 26,236 Reputation points

2021-05-26T22:23:33.78+00:00

Thank you for letting me know length isn't the problem, @Connor Hazen .

I would like to pass your case into the care of my colleague @ChaitanyaNaykodi-MSFT . Azure Functions falls under his area of expertise.

Answer 1

ChaitanyaNaykodi-MSFT 27,471 Microsoft Employee Moderator

Hello @Connor Hazen , Thank you for providing the details above. While looking at the backend, I think the issue behind intermittent 404 errors might be related to the synchronous implementation of the code. As Python is a single-threaded runtime, a host instance for Python can process only one function invocation at a time by default. For applications that process a large number of I/O events and/or is I/O bound, you can improve performance significantly by running functions asynchronously. It will be helpful you could go through this Async section of the Scale and Performance documentation for Python Function Apps and see if making the code Async helps in getting rid of the issue.
Let me know if there are any additional concerns. Thank you!

Connor Hazen 1 Reputation point

2021-05-31T16:29:30.177+00:00

Thank you for the advice, this is certainly a change we will implement. However, I am doubtful as to it being the only issue. For example, we yesterday had an invocation fail where the previous run was over an hour before. How would the single threaded nature of python cause this issue when there is only one run occurring, or it is the first run in a cluster.
Connor Hazen 1 Reputation point

2021-05-31T18:10:48.587+00:00

From my own digging, this seems quite similar to this https://github.com/Azure/azure-functions-host/issues/5149

There are no concurrent active runs for some of the failures, there is no timeout issue here, and it does not appear to be a scale issue.

Furthermore, I have made the function asynchronous now, this did not solve the issue.

Finally, it's also important to note that these failures do not show up on the monitor panel of the the function app. They are not failed runs or internal errors, they simply do not make it to the function app. Hence why the error link above seems related.
Mitchell Bennett 1 Reputation point

2021-06-08T03:01:36.83+00:00

Did you find a solution to this?

Just wanted to say that I am experiencing this issue also. The issue description is basically a perfect match with the problem I am having except I'm using the HTTP trigger.

Answer 2

Connor Hazen 1

I solved this problem, I believe, in a roundabout manner. Still not sure what was causing it. However, I moved all of my code to a new function, meaning I made a new function app, made a new function in visual studio code, and then copied over the scripts(I left the new .vscode directory etc. - essentially everything expect the python files I had written was replaced by the new function I had made).

UPDATE: spoke too soon. still experiencing issues.

Answer 3

Mitchell Bennett 1

I have created a support ticket for this issue and have been told that the problem has been identified by Microsoft engineers and they are working on an "enhancement" for it.

In the mean time what I have done to work around this issue is to set the maximum number of retry attempts and the retry interval on each Azure Function activity in my pipeline.

Hopefully this helps and the issue gets resolved soon!

Connor Hazen 1 Reputation point

2021-06-24T20:39:29.203+00:00

Are you by chance using python 3.9? I have a suspicion this may be the cause.
Mitchell Bennett 1 Reputation point

2021-06-24T22:03:04.82+00:00

I have tried using both Python 3.9 and 3.6 and I still had the same problem.
Mitchell Bennett 1 Reputation point

2021-07-01T04:12:57.327+00:00

I have received feedback from Microsoft employees about this issue and their position, at time of writing, is that the 404 issue (outlined in the problem description) is expected due to the cold-start limitation of the Consumption Plan for Azure Functions. They report that switching to the Premium Plan, which contains dedicated resources, is supposed to prevent this issue from happening. If the Premium Plan is undesirable then they recommend setting the number of retry attempts and the retry interval on the Azure Function activity - as I mentioned in my comment above.
Taylor Teske 1 Reputation point

2021-07-14T17:38:45.54+00:00

Out of curiosity, with setting the retry attempts and interval, have you still encountered any 404s? I stumbled upon this recent thread as we seem to be experiencing intermittent 404s and tried implementing a max retry to 3 and delay interval to 10 seconds.

Answer 4

Christian Schreuder 11

Hi, if you restart your Function App does the error go away for a few hours before it comes back?

Answer 5

dthunn 1

At my work we are experiencing the exact same issue, about 1 / 10 requests results in a 404, there is no clear way to fix this problem.

Christian Schreuder 11 Reputation points

2021-07-23T15:58:02.553+00:00

Upgrading the function app to premier tier, where it doesn't have this cold start issue, works as a temporary fix.

Share via

ADF calling Azure Function: '404 not found' exception once out of every 10 runs (after 2 seconds)

5 answers

Your answer