Call Databricks notebook from azure data factory | Could not parse request object

Question

Call Databricks notebook from azure data factory | Could not parse request object

Lokesh 211

We are getting the following error while executing the data bricks (DBX) notebook via azure data factory (ADF).

Could not parse request object: Expected both 'key' and 'value' to be set on elements in the field 'base_parameters' at [Source: (ByteArrayInputStream); line: 1, column: 2802] at [Source: java.io.ByteArrayInputStream@4491e40b; line: 1, column: 2802]

This occurs during the debug mode when you have a parameter being passed from ADF to DBX and the parameter is being received through a SQL query or stored procedure.
If the parameter value is null, data bricks notebook fails to run with the above error message.

It seems that parameter values don't honor lower case and if you hardcode the value to NULL, the issue disappears.

We have seen this bevaiour only while debugging the notebook

1 answer

Your answer

Answer 1

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

Hello @Lokesh ,

Thanks for the question and using MS Q&A platform.

It seems that parameter values don't honor lower case and if you hardcode the value to NULL, the issue disappears. - I doubt lower/upper case is a problem here. As per my understanding when you are passing NULL as parameter value, you are passing a hardcoded valid string value to your parameters and hence it is not failing and when the auto generated parameter value is null (empty) that means it is an empty string (no value) and hence it is failing.

Here is an existing TSG which calls out this issue and the possible resolution: Azure Databricks Error code: 3202

Message: Could not parse request object: Expected 'key' and 'value' to be set for JSON map field base_parameters, got 'key: "..."' instead.

Cause: Authoring error: No value provided for the parameter.

Recommendation: Inspect the pipeline JSON and ensure all parameters in the baseParameters notebook specify a nonempty value.

Please ensure that your pipeline passes a non-empty value to the baseParametres notebook, and it should help resolve the issue.

Hope this info helps

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Lokesh 211 Reputation points

2022-11-17T12:30:04.197+00:00

we are calling a stored procedure in the lookup activity prior to calling the notebook
one of the values returned from this stored procedure is null (received as null).

This is causing issues.
Lokesh 211 Reputation points

2022-11-17T12:31:37.597+00:00

seems like during debugging lower null throws as error!

Please ask if you need more information.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-11-17T22:00:02.073+00:00
Thanks for your response @Lokesh . Would you please save the Lookup output response JSON Payload to a file and rename extension to .txt and attach it here by replacing/masking actual data with dummy data expect for the field that is having null value and creating the issue?

Are you referring to JSON null as below:

[{ "name": "John", "surname": null }, { "name": "Jane", "surname": "Doe" }]

Or as lower-case string value (not empty/null) as below:

[{ "name": "John", "surname": "null" }, { "name": "Jane", "surname": "Doe" }]
Lokesh 211 Reputation points

2022-11-18T09:07:58.18+00:00

its the first one.

Please see the output below
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-22T23:00:53.833+00:00

Hello @Lokesh ,

As per your statement, you are passing json null values as parameters to your notebook which is why it is throwing error as they are treated as No value provided for the parameter (i.e., empty value).

To avoid this issue, please ensure to pass a valid (non-empty) parameter value to your notebook.
Either you can tweak your stored procedure to check for json null values and do replace with a valid default value or your can also filter those parameters with json null value and only pass the valid parameters values to you ADB notebook.

Ref doc: Error code: 3202

Hope this info helps.

Share via

Call Databricks notebook from azure data factory | Could not parse request object

1 answer

Your answer