Single Pyspark notebook to call multiple notebooks through parameters

Bathini, Shirish 0 Reputation points
2024-04-18T09:43:02.48+00:00

I have one pyspark notebook1 which loads one ERP system (ex:A ) , another notebook loads one more ERP system ( ex: B) .

I want to create another notebook to call above 2 notebooks through parameters ..

the notebook should ask parameter to enter ERP , when you pass ERP to A , it should execute notebook1 , if it is B , it should execute notebook2 ..

Please help in achieving this usecase . I tried with %run .. but unable to pass parameters ..

Thank You

Shirish

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,477 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Smaran Thoomu 10,880 Reputation points Microsoft Vendor
    2024-04-23T11:29:59.9666667+00:00

    Hi @Bathini, Shirish

    Thanks for the follow-up question.

    You are correct that dbutils.widgets are not supported in Synapse work environment. Instead, I think you can use the built-in Python input() function to prompt the user to enter the ERP system. Here's an example:

    erp_system = input("Enter the ERP system (A or B): ")
    

    Regarding the error you are getting with %run "/path/to/notebook1", it's possible that the path to the notebook is incorrect. Please make sure that you are providing the correct path to the notebook.

    To pass parameters to the notebooks in Synapse, you can define variables in the new notebook and pass them as arguments to the notebooks being executed using the %run magic command. Here's an example:

    if erp_system == "A": 
    	notebook_path = "/path/to/notebook1" notebook_params = "--param1 value1 --param2 value2" 
    	elif erp_system == "B": 
    	notebook_path = "/path/to/notebook2" notebook_params = "--param1 value1 --param2 value2" 
    else: 
    	print("Invalid ERP system")
    %run $notebook_path $notebook_params
    

    This code defines notebook_path and notebook_params based on the value of erp_system, and then passes them as arguments to the %run magic command.

    I hope this helps! Let me know if you have any further questions.

    1 person found this answer helpful.
    0 comments No comments