Unexpected error on passing json value in azure data factory

Question

Unexpected error on passing json value in azure data factory

Ankit Kumar 71

0

I am calling an Azure Function(HTTP trigger) in Azure Data Factory and the body is coming from a lookup activity (@activity('Lookup1').output.value) which is something like this

"body":[
        {
            "BaseObject": "2|03|01|01",
            "BaseObjectDescription": "Cent",
            "CreateDate": "30.09.2021"
        },
        {
            "BaseObject": "9|03|01|01",
            "BaseObjectDescription": "Pent",
            "CreateDate": "30.09.2021"
        }]

The above json when passed as body to Azure function activity, I get error as

"Unexpected character encountered while parsing value: S."

But if I hardcode and pass the same value in body of Azure function in azure data factory, I get the output. When I hardcode it, I see in debug mode in Azure Data factory the body is passed as something like this

"body": "[\n        {\n            \"BaseObject\": \"02|03|01|01\",\n            \"BaseObjectDescription\": \"Cent\",\n            \"CreateDate\": \"30.09.2021\" },\n        {\n            \"BaseObject\": \"04|03|01|01\",\n            \"BaseObjectDescription\": \"Pent",\n            \"CreateDate\": \"21.09.2021\",\n },\n]"

So question is how do I change the json I am getting from lookup activity to something like above so that my Function recognizes this as body in Azure Data Factory

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-01-07T18:06:55.57+00:00

HI @Ankit Kumar ,

Could you please share screenshots of your Azure function request body when its hard coded and also when its passed dynamically from lookup activity output?

So that we can have quick check if something is missing while passing them dynamically.

2 answers

Your answer

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-01-07T18:06:55.57+00:00

HI @Ankit Kumar ,

Could you please share screenshots of your Azure function request body when its hard coded and also when its passed dynamically from lookup activity output?

So that we can have quick check if something is missing while passing them dynamically.

Answer 1

Niels 236

Both outputs are the same, but in the last one you are seeing line break and escape characters. \n means a line break and \" means that the quote is escaped, because the whole string is enclosed in quotes.

You should be focusing on where the error "Unexpected character encountered while parsing value: S." is coming from.

Ankit Kumar 71 Reputation points

2022-01-05T13:35:36.577+00:00

Hi @Niels

but both the bodies are same and I am passing it once as a output to lookup activity and other time it is directly hardcoded which Datafactory automatically does with line break and escape character. If I can convert the output coming from Lookup to something like I get with directly hardcoding, it should resolve.

Note: I tried passing the same json from lookup to azure function uri in POSTMAN and it works fine. so something is happening on ADF by its own
Niels 236 Reputation points

2022-01-05T14:49:47.583+00:00

The json is probably fine. As you said yourself, the json works fine when using Postman. So don't stare yourself blind at the json, but look for the error in another place. For some reason ADF is trying to parse the value "S." and you get the error. Maybe it's not what you think it is, that is causing the problem. Take a step back and trace the cause error message description. And look at other values that might contain "S.".
Ankit Kumar 71 Reputation points

2022-01-05T20:45:24.307+00:00

Other than JSon, its only the function name which has S in it. I dont know where else to look
Ankit Kumar 71 Reputation points

2022-01-05T21:18:08.97+00:00
So to check if something to do with function name , I created another function with no "S" in the function name and I get the same error. The error code i get in ADF is as below but there is no further details in the troubleshooting guide

Error details Error code 3608 Troubleshooting guide Failure type User configuration issue Details Call to provided Azure function 'Function1' failed with status-'InternalServerError' and message - 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'. Source Pipeline pipeline_with_dataflow
Niels 236 Reputation points

2022-01-06T09:00:10.397+00:00

For Azure Functions error 3608 InternalServerError might mean there's something wrong with the body or passed arguments of the call. If you have application insights active on the web app service for Azure Functions, check there for the exception.
Ankit Kumar 71 Reputation points

2022-01-06T09:16:45.67+00:00

When I went to logs in Azure Function, this is what I see. The one success is from calling it form POSTMAN. What argument is being talked about here_ I have an url and the Body and method is POST
Ankit Kumar 71 Reputation points

2022-01-06T09:46:26.337+00:00

One thing I noticed on the error message in ADF, is when creating the Linked service, I enter the Function URL manually like this below

I get error as "Call to provided Azure function 'Function1' failed with status-'NotFound' and message - 'Invoking Azure function failed with HttpStatusCode - NotFound.'."

But when i edit the linked service and select the function URl from azure subscription like this

I get error in my function as Internal Server Error and in logs its Unexpected character encountered while parsing value: S.
The Function URL in first case is the URL I copied from Get Function URL in the azure and it is of this format "https://afmodel.azurewebsites.net/api/Function1". Is there anything wrong here?
Niels 236 Reputation points

2022-01-06T10:19:39.727+00:00

The function app URL should only be "https://afmodel.azurewebsites.net" without the function name. Selecting the Azure Function from the Azure subscription is the best way to go. And you Azure Function action should contain the function name you want to call.

The error message from the function states: "The argument 'length' is smaller than the minimum of 1". Does your function have an argument named "length"? If yes than this argument should have a value of 1 or higher passed though.

(BTW, for security reasons it's best to select managed identity for authentication method instead of anonymous for the linked service, but that's out of the scope of this answer.)
Ankit Kumar 71 Reputation points

2022-01-06T10:35:12.343+00:00

The endpoint has following header in Postman and content-length is calculated on call. I have not specifically added Content-length here
Niels 236 Reputation points

2022-01-06T10:50:46.077+00:00

For an anonymous function call, that look alright. Maybe the exception is thrown by code within your function? I can't think of anything else.
Ankit Kumar 71 Reputation points

2022-01-06T10:57:09.94+00:00

if that would have been the case, it should be with Postman or with hardcoded body value in ADf as well.
Niels 236 Reputation points

2022-01-06T11:42:41.257+00:00

That's true. Maybe ADF is not passing the body parameter that you think it is doing. Have a look at that.

If you want to go hardcore, you might want to debug the Azure Function remotely in VS to see what is passed to the function. Check this article: https://www.linkedin.com/pulse/remote-debug-your-azure-function-app-v2-visual-studio-krish
Ankit Kumar 71 Reputation points

2022-01-06T12:09:54.237+00:00

I am copying the body parameter from ADF to Postman and it works fine. Even I am hardcoding the body parameter which I get from Lookup and it works fine.
Niels 236 Reputation points

2022-01-06T16:46:47.217+00:00

We're you able to solve the problem? If not, how can I help you?
Ankit Kumar 71 Reputation points

2022-01-06T21:25:05.033+00:00

no I have not been able to solve it ...I dont know if this is an issue with Function v4 version being called from ADF? I am going to recreate a simple function and call it from ADF and see how it goes and then add my code line by line.
Ankit Kumar 71 Reputation points

2022-01-07T08:11:59.09+00:00

another day another finding.. The issue is not with body. It seems my function api should work with an empty body and it does in postman but in ADF it fails with same error
Ankit Kumar 71 Reputation points

2022-01-07T09:50:45.337+00:00

I have another error message this time which is something like this
System.ArgumentNullException : Value cannot be null. (Parameter 's')at System.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)at System.Convert.FromBase64String(String s)at test_app.Function1..ctor

Ankit Kumar 71

and in my code I have a constructor which is something like this where I have a Servicecrendential being converted to base64. Is that causing the error?

namespace test_app
{
    public class Function1
    {
        readonly GetConnectionDetails connection = new GetConnectionDetails();
        // private readonly string _baseURL;
        private readonly string _iapId;
        private readonly ServiceAccountCredential _saCredential;
        private TokenResponse _token;
        static readonly HttpClient client = new HttpClient();


public Function1()
        {

            var base64ServiceAccount = connection.base64ServiceAccount;
            _iapId = connection.iapId;
            _saCredential = ServiceAccountCredential.FromServiceAccountData(new MemoryStream(System.Convert.FromBase64String(base64ServiceAccount)));
            client.DefaultRequestHeaders.Add("IMBuild", Environment.GetEnvironmentVariable("VERSION_HEADER") ?? "dev");
        }

//my other methods in this class
}

public class GetConnectionDetails
    {

        public string baseUrl = Environment.GetEnvironmentVariable("baseUrl");
        public string iapId = Environment.GetEnvironmentVariable("iapId");
        public string base64ServiceAccount = Environment.GetEnvironmentVariable("base64ServiceAccount");

    }

Ankit Kumar 71 Reputation points

2022-01-07T11:50:41.86+00:00

i removed base64 value from environment and hardcoded in for now. I get different error "Executed 'Function1' (Failed, Id=22e0573f-6300-46e8-9ef1-30801f3a7736, Duration=276ms)The argument 'length' is smaller than minimum of '1' (Parameter 'length')"

Answer 2

Ankit Kumar 71

For those who face this issue I solved it by putting the output in a variable and then passing the variable as a string in my Function activity body. This resolved the issue. Thanks

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-01-12T16:38:57.157+00:00

Hi @Ankit Kumar - Thank you for sharing your workaround. Kindly consider marking it as accepted answer by hitting Accept Answer button. Accepted answers helps community as well.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-01-19T13:21:15.917+00:00

Hi @Ankit Kumar - Please consider marking it as Accepted Answer. Accepted answers helps community as well.

Share via

Unexpected error on passing json value in azure data factory

2 answers

Your answer