Facing issues while running multiple notebook in a single notebook via pipeline and the same notebook runs fine when it is ran without pipeline

Question

Error data_quality { "errorCode": "6002", "message": "MagicUsageError: Cannot read notebook common_variables. The possible reason is that the notebook doesn't exist. More details please visit https://go.microsoft.com/fwlink/?linkid=2173018\n--> PartnerException: Exception of type 'Microsoft.Analytics.SynapseNotebookService.Infrastructure.Exceptions.ClientExceptions.PartnerException' was thrown.", "failureType": "UserError", "target": "data_quality", "details": [] }

Answer 1

Hi @Abhinesh Kumar Lal Karn

Welcome to Microsoft Q&A platform and thanks for posting your question here.

To troubleshoot the issue, you should first check if the common_variables notebook exists in the specified location. If it doesn't exist, create the notebook and try running the pipeline again. This is because the error message suggests that the notebook cannot be found, so it's important to ensure that it exists in the specified location. Once you've created the notebook, try running the pipeline again to see if the issue is resolved.

If the notebook exists but the issue still occurs, the next step is to verify that the notebook path is correct. You can do this by checking the pipeline activity configuration and ensuring that the path is specified correctly. Check if the pipeline activity is using the correct notebook kernel. If the kernel is not specified correctly, it can cause issues while running the notebook. 

Make sure that the pipeline activity has the necessary permissions to access the notebook and its dependencies. Also, try running the notebook in interactive mode to check if it runs without any errors. If it runs smoothly, the issue might be with the pipeline configuration.

Relevant thread
https://stackoverflow.com/questions/73979005/synapse-execute-the-magic-command-run-notebook-from-pipeline

I hope this helps you resolve your issue. Let me know if you have any further questions or concerns.

 

Answer 2

You will experience this issue, in case if you have not published the notebook which you are calling from another notebook while running through the pipeline