Synapse Pipeline Notebook Dynamic Content. Error 2011: "Processed HTTP request failed."

Question

Issue seems very related to the one described here: https://learn.microsoft.com/en-us/answers/questions/708155/
Unfortunately, this issue did not resolve when refreshing Synapse studio and hence I am still stuck.

The case:
We are implementing a generic Synapse pipeline to call Synapse notebooks using Dynamic Content for naming. The pipeline will get the Notebook name as input parameter and pass this through to the Notebook activity. In addition, a parameter object is passed, which is a JSON object dumped to a string. The notebook parses the JSON again and uses it for logic configuration.

The pipeline does not do anything other than calling this Notebook, given the input parameters. Its definition can be seen below:

Upon calling the pipeline with an existing Notebook name, regardless of the parameter object passed through, we are seeing the following error after mere seconds:

   {  
       "errorCode": "2011",  
       "message": "Processed HTTP request failed.",  
       "failureType": "UserError",  
       "target": "Execute Notebook",  
       "details": []  
   }  

When calling the notebook directly (without dynamically specifying its name), with or without the parameter object works and gives the desired results.

What is going on here and who can help me with a solution?

If deemed useful, here's a copy of the pipeline definition JSON:

   {  
       "name": "Execute Notebook",  
       "properties": {  
           "activities": [  
               {  
                   "name": "Execute Notebook",  
                   "type": "SynapseNotebook",  
                   "dependsOn": [],  
                   "policy": {  
                       "timeout": "7.00:00:00",  
                       "retry": 0,  
                       "retryIntervalInSeconds": 30,  
                       "secureOutput": false,  
                       "secureInput": false  
                   },  
                   "userProperties": [],  
                   "typeProperties": {  
                       "notebook": {  
                           "referenceName": {  
                               "value": "@pipeline().parameters.NotebookName",  
                               "type": "Expression"  
                           },  
                           "type": "NotebookReference"  
                       },  
                       "parameters": {  
                           "parameterObject": {  
                               "value": {  
                                   "value": "@pipeline().parameters.ParameterObject",  
                                   "type": "Expression"  
                               },  
                               "type": "string"  
                           }  
                       },  
                       "snapshot": true,  
                       "sparkPool": {  
                           "referenceName": "Spark",  
                           "type": "BigDataPoolReference"  
                       }  
                   }  
               }  
           ],  
           "parameters": {  
               "NotebookName": {  
                   "type": "string"  
               },  
               "ParameterObject": {  
                   "type": "string"  
               }  
           },  
           "annotations": []  
       }  
   }  

Answer 1

Hello @BoyR ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is "why do I get the ambiguous error message only when I try to run notebooks selected by dynamic content?"

I have reproduced the error message, with slight variation on cause(s). My initial hypothesis for the cause was whether the notebook had been published or not. Things got a bit more complicated, so I am sharing my test results below. The factors I am testing include notebook type, published/unpublished, committed/uncommitted (applies to repo mode only), whether a spark pook has been selected or not (Select...), and whether the notebook is selected manually or dynamically.
The spark pool selection so far seems to have little if no effect.

If your experience corroborates or conflicts with my test results let me know.

----------

Test findings:

I get the error message when...

• the dynamically selected notebook name wrong ("ThisNotebookDoesNotExist")
• the dynamically selected pyspark notbook is unpublished (live mode)
• the manually selected unpublished SQL notebook (repo mode)
• the manually selected uncommited and unpublished pyspark notebook (repo mode)

Worked when:

• manually selected unpublished but committed pyspark notebook
• the dynamically selected pyspark notebook has been previously published (Live mode)

Thanks
Martin

------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how