ADF Function Activities fail to return response body with 400/500 status codes

Question

ADF Function Activities fail to return response body with 400/500 status codes

Scoot-3223 71

Why does a ADF pipeline's function activity omit the response body when a 400/500 status code is returned, while the response body is returned with the same function is run directly from a function app?

I have a simple python test function, shown below, that executes as expected as a standalone function. The 500 status_code AND the response body are returned. However, when this function is executed in a data factory pipeline function activity, the payload is dropped.

I'd like to use 500 status codes to enable error handling in my function based pipelines, and to capture a raised exception which is excluded here for simplicity.

Thanks,
Scott

Simple test function:

import azure.functions as func

def main(req: func.HttpRequest) -> func.HttpResponse:

return func.HttpResponse("Exception was raised", status_code=500)

P.S. In researching this ADF anomaly I found this which confirms what I see in the ADF: https://stackoverflow.com/questions/61949996/why-is-azure-api-management-hides-response-body-for-status-codes-4-and-5. This seems like the issue but I have no idea how to test a solution.

While this link confirms what I see when running a stand alone function: https://learn.microsoft.com/en-us/answers/questions/509540/why-do-azure-functions-report-success-if-http-500.html

Answer 1

MartinJaffer-MSFT 25,711

Hello @Scoot-3223 and welcome to Microsoft Q&A.

Thank you for such a well written question. I see you have done some research already.

I used your simple test function to repro the issue, with 1 alteration; I added my name to the message so I can differentiate from some standard message. However, I saw the exception message was returned! I tested this with both Web Activity and Azure Function Activity in ADF. I see it in "Error details" rather than "Output". I tested with both a debug run and a "trigger now".

The Function Activity's, error:

Call to provided Azure function 'FailHttp' failed with status-'InternalServerError' while invoking 'GET' on 'https://failfunc.azurewebsites.net' and message - 'Martin's Exception was raised'.

func-error-snapshot

The Web activity's error:

Martin's Exception was raised

web-error-snapshot

I know you didn't say anything about Web activity, but I like to use it for comparison.

Since our experiences differed so wildly, I'd like to verify whether you were checking the output or the errors for your response body. ... ohh, now I get what you are asking! You are asking about the design decision to make the same message component have different jsonpath depending upon response code.
Yes, technically, the response body is the response body no matter the error code from a raw point of view.

However having them different protects code assuming success from running on error message instead of data.

Also, with the design decision to make status codes 400/500 result in an activity failure, we want the Function activity to behave similar to how other activities behave on failure/error behavior. Make the failure/error output align across different activities. To this end, we need a error message or other detail to pass along with the code. While we could put a standard explanation of the status code, using the response body is richer and respects the endpoint's intent more. Where else could the error message come from?

Did I answer your question, or are you really missing the message even in the error class? I wonder if there is a character limit to that. Stack traces can get very large.

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Scoot-3223 71 Reputation points

2022-11-18T01:03:11.81+00:00
Hi Martin,

Your response is super helpful. I was aware of the Error component but didn't know how to retrieve it. When we look at the function response all we see just the payload with billing information

Since you mentioned an error object, I updated the activity expression I created for use in the webhook notification to be the following and it worked! The webhook is linked to the upstream activity via ON SKIP, and it now captures any exception upstream in the pipeline:

@markus.bohland@hotmail.de ('{"text":"',
activity('logBlob').error.errorCode,
' ',
activity('logBlob').error.message,'"}')

It just took me a bit of trial an error to come up with error.errorCode and error.message.

Now, what I'm finding is that we can only return exception messages for exceptions that are explicitly caught, like this:

except Exception as err:
return func.HttpResponse(f"HttpLogBlob encountered exception from Scott {err}",
status_code=500)

However, for an uncaught exception, then we loose the ability to report on the exception message. For example, if my test function just blindly raises an exception as shown here:

raise Exception("HttpLogBlob threw an exception as requested in the input") #return func.HttpResponse("Scott's function errored out", status_code=500). Not needed as the Exception causes an immediate return.

continued in second comment...
Scoot-3223 71 Reputation points

2022-11-18T01:06:30.87+00:00

Continuation of first comment, due to character limit...

The error Json that's passed into the webhook activity looks like:

{
"url": "retracted",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": "{\"text\":\"3608 Call to provided Azure function 'httpLogBlob' failed with status-'InternalServerError' while invoking 'POST' on 'https://scspyfunctionapp.azurewebsites.net' and message - 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'.\"}"
}

So you can see that for uncaught exceptions, the exception message is lost. And unfortunately this greatly reduces the value of sending exception notifications, we learn a function failed but we don't immediately see the cause. Granted we'd likely have to look at the app insight or monitor logs anyhow to debug the issue.

With all that said, do you know of any way to capture the exact exception message for uncaught exceptions via an expression that references the failed activity and its error json?

Thanks,
Scott
MartinJaffer-MSFT 25,711 Reputation points

2022-11-29T19:06:09.627+00:00
I have a suggestion to ease that sort of development in the future.

In a debug run, I set a variable to @{activity('logBlob')} This captures everything about the activity, input, output, error, and more. I then copy it to a text editor, unescaped the stuff, and then run a json formatter to clean it up. This then lets me see all properties of the activity run.

My apologies for making assumption that you knew about the errors.

While on the topic, there is a useful operator. The null-safe operator. You probably noticed when the activity succeeds, the error portion does not exist, and this causes issues when you have an expression meant to retrieve the error. This is where the null-safe operator comes into play.

activity('logBlob').?error.?errorCode

This makes it so no error is throws when errorCode does not exist. Instead a null value is returned. Or maybe empty string. The usefulness is seen in

coalesce( activity('logBlob').?error.?stuff , activity('logBlob').?output.?stuff )

coalesce returns the first non-null value. Since the existence of output and error are mutually exclusive, we can safely get the body in both success and failure.

sorry for the delay, @Scoot-3223 , I caught that nasty respiratory illness going around.
MartinJaffer-MSFT 25,711 Reputation points

2022-11-29T19:44:18.113+00:00

There is a subtlety I'd like to introduce you to.

However, to answer your question, the Data Factory only knows what the Function responds with. Data Factory does not have a window into the Function app beyond what you provide. Before you think too harshly, imagine the Function app was a website, and ADF activity as a web browser. Would you want the whole internal stack exposed? (For dev/testing purposes some would say yes, but the point here is to reframe what is safe/reasonable.)

The subtlety is the difference between a program failing, and the service running the program failing. Uncaught exceptions look like the service running the program failing, while caught and handled exceptions look like the program gracefully failing.
The difference is to know whether something went wrong on Azure, or something went wrong on your code.

So when the Function is unable to complete, and thus your code unable to send a reply, a default reply is created by the Function service itself. If there is a way to alter the Function service to send the whole stack, I am unaware of it. I'll ask a colleague who works with Functions.

So no, ADF itself can't get uncaught exceptions raw.

However, if you wrap everything in a try, and use a base exception class to catch all stuff, you should be able to achieve your goal, I think. It has been a while since I went deep into exception classes. @Scoot-3223
Scoot-3223 71 Reputation points

2022-12-01T19:10:46.24+00:00

Martin, Again, your reply is super helpful. I appreciate the extra info on how to capture activity output into a variable. Also the rationale for the differences in caught versus uncaught exceptions. It makes perfect sense now.

Thanks again.

ADF Function Activities fail to return response body with 400/500 status codes

0 additional answers

Activity