Call Databricks notebook from azure data factory | Could not parse request object

Lokesh 211 Reputation points
2022-11-16T09:33:19.293+00:00

We are getting the following error while executing the data bricks (DBX) notebook via azure data factory (ADF).

Could not parse request object: Expected both 'key' and 'value' to be set on elements in the field 'base_parameters' at [Source: (ByteArrayInputStream); line: 1, column: 2802] at [Source: java.io.ByteArrayInputStream@4491e40b; line: 1, column: 2802]

This occurs during the debug mode when you have a parameter being passed from ADF to DBX and the parameter is being received through a SQL query or stored procedure.
If the parameter value is null, data bricks notebook fails to run with the above error message.

It seems that parameter values don't honor lower case and if you hardcode the value to NULL, the issue disappears.

We have seen this bevaiour only while debugging the notebook

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,080 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,187 questions
0 comments No comments
{count} vote

1 answer

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee
    2022-11-17T00:09:40.853+00:00

    Hello @Lokesh ,

    Thanks for the question and using MS Q&A platform.

    It seems that parameter values don't honor lower case and if you hardcode the value to NULL, the issue disappears. - I doubt lower/upper case is a problem here. As per my understanding when you are passing NULL as parameter value, you are passing a hardcoded valid string value to your parameters and hence it is not failing and when the auto generated parameter value is null (empty) that means it is an empty string (no value) and hence it is failing.

    Here is an existing TSG which calls out this issue and the possible resolution: Azure Databricks Error code: 3202

    Message: Could not parse request object: Expected 'key' and 'value' to be set for JSON map field base_parameters, got 'key: "..."' instead.

    Cause: Authoring error: No value provided for the parameter.

    Recommendation: Inspect the pipeline JSON and ensure all parameters in the baseParameters notebook specify a nonempty value.

    Please ensure that your pipeline passes a non-empty value to the baseParametres notebook, and it should help resolve the issue.

    Hope this info helps


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how