DataBricks assertion when calling a notebook from another notebook on high-concurrency cluster with passthrough enabled

Question

in DataBricks premium:

getting this assertion failure when calling a notebook from another notebook on high-concurrency cluster with passthrough enabled

com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: assertion failed: Attempted to set keys (credentials) in the extraContext, but these keys were not in the set of valid keys: {commandResultJsonMaxBytes, displayRowLimitV2, notebook_path, thresholdForStoringInDbfs, enableStoringResultsInDbfs, api_url, aclPathOfAclRoot, displayByteLimitV2}

very easy to reproduce, started happening on Jan 13, 2022, a day before all was working fine
code in python cell:

dbutils.notebook.run('second', 0) <= this will err out on high-concurrency cluster with passthrough enabled (if passthrough is not enabled it works fine)

Answer 1

Hi @Maciej G ,

Thank you for posting query in Microsoft Q&A Platform.

In order to reproduce your scenario, I created Azure data bricks Premium account and then created high-concurrency cluster with passthrough enabled.
Created two notebooks which are notebook1 & notebook2. From notebook2 to I tried to run notebook1 using dbutils.notebook.run() function. It was running notebook good to me. Kindly check below image.

I followed up on similar issue in databricks community as well. It seems fix implmented from product side for this. So now, we should be good here. Could you please try from your end too and confirm? Below is the link of databricks community discussion.
https://community.databricks.com/s/question/0D73f000001Z9Oz/detail

Hope this helps. Please let us know how it goes.

------------------

Please consider hitting Accept Answer. Accepted answers helps community as well.