Passing dynamic parameter to %Run command in Synapse Notebook

Question

So Im doing a test to see if we can dynamically send parameters from a parent notebook to a child notebook

Child Notebook:

from pyspark.sql import SparkSession
from notebookutils import mssparkutils


print(f"Storage Account Name: {storage_account_name}")
print(f"Container Name: {container_name}")
print(f"Data Load Process Name: {DataLoadProcess_name}")
print(f"Latest Batch Number: {LatestBatchNumber_value}")
print(f"Latest Batch Load Start Date: {LatestBatchLoadStartDate_value}")
print(f"Latest Batch Load End Date: {LatestBatchLoadEndDate_value}")

Parent Notebook:

storage_account_name = "storageaccountname"
container_name = "container"
DataLoadProcess_name = "ProcessName"
LatestBatchNumber_value = None
LatestBatchLoadStartDate_value = "2024-10-03T10:59:27.695108Z",
LatestBatchLoadEndDate_value = None

# Construct the parameters dictionary dynamically
params = {
    "storage_account_name": storage_account_name,
    "container_name": container_name,
    "DataLoadProcess_name": DataLoadProcess_name,
    "LatestBatchNumber_value": LatestBatchNumber_value,
    "LatestBatchLoadEndDate_value": LatestBatchLoadEndDate_value
}

So I've tried:

%run "HouseKeeping/WZ_NB_DataLoadProcessMerge" {params}

and i get this error

MagicUsageError: Cannot parse notebook parameters. More details please visit https://go.microsoft.com/fwlink/?linkid=2173018 --> JsonReaderException: Invalid JavaScript property identifier character: }. Path '', line 1, position 7.

I've also tried the below:

mssparkutils.notebook.run("HouseKeeping/WZ_NB_DataLoadProcessMerge", 600, params)

and got this error:
NameError: name 'storage_account_name' is not defined

Is it possible to dynamically pass parameters from a parent notebook to a child notebook?

Answer 1

In Azure Synapse Analytics, passing dynamic parameters from a parent notebook to a child notebook can be accomplished, but it requires careful formatting, especially when using the %run command or the mssparkutils.notebook.run function.

1. Using %run Command

The %run command does not support passing parameters as a dictionary directly. Instead, you need to construct the command as a formatted string. Here's how you can do it:

Parent Notebook (SQL)

storage_account_name = "storageaccountname"

container_name = "container"

DataLoadProcess_name = "ProcessName"

LatestBatchNumber_value = None

LatestBatchLoadStartDate_value = "2024-10-03T10:59:27.695108Z"

LatestBatchLoadEndDate_value = None

# Construct the parameters string for the %run command

params_string = f"""

%run "HouseKeeping/WZ_NB_DataLoadProcessMerge" {storage_account_name} {container_name} {DataLoadProcess_name} {LatestBatchNumber_value} {LatestBatchLoadStartDate_value} {LatestBatchLoadEndDate_value}

"""

# Run the child notebook with parameters

exec(params_string)

Child Notebook (Python)

You will need to accept parameters in the child notebook using the notebookutils package to extract them:

from pyspark.sql import SparkSession

from notebookutils import mssparkutils

# Fetch parameters passed from the parent notebook

storage_account_name = mssparkutils.notebook.getContext().get("storage_account_name")

container_name = mssparkutils.notebook.getContext().get("container_name")

DataLoadProcess_name = mssparkutils.notebook.getContext().get("DataLoadProcess_name")

LatestBatchNumber_value = mssparkutils.notebook.getContext().get("LatestBatchNumber_value")

LatestBatchLoadStartDate_value = mssparkutils.notebook.getContext().get("LatestBatchLoadStartDate_value")

LatestBatchLoadEndDate_value = mssparkutils.notebook.getContext().get("LatestBatchLoadEndDate_value")

print(f"Storage Account Name: {storage_account_name}")

print(f"Container Name: {container_name}")

print(f"Data Load Process Name: {DataLoadProcess_name}")

print(f"Latest Batch Number: {LatestBatchNumber_value}")

print(f"Latest Batch Load Start Date: {LatestBatchLoadStartDate_value}")

print(f"Latest Batch Load End Date: {LatestBatchLoadEndDate_value}")

2. Using mssparkutils.notebook.run

When using mssparkutils.notebook.run, make sure to pass the parameters correctly in a dictionary and ensure that all variables are defined in the parent notebook:

Parent Notebook (SQL)

storage_account_name = "storageaccountname"

container_name = "container"

DataLoadProcess_name = "ProcessName"

LatestBatchNumber_value = None

LatestBatchLoadStartDate_value = "2024-10-03T10:59:27.695108Z"

LatestBatchLoadEndDate_value = None

# Construct the parameters dictionary dynamically

params = {

    "storage_account_name": storage_account_name,

    "container_name": container_name,

    "DataLoadProcess_name": DataLoadProcess_name,

    "LatestBatchNumber_value": LatestBatchNumber_value,

    "LatestBatchLoadStartDate_value": LatestBatchLoadStartDate_value,

    "LatestBatchLoadEndDate_value": LatestBatchLoadEndDate_value

}

# Pass the parameters to the child notebook

mssparkutils.notebook.run("HouseKeeping/WZ_NB_DataLoadProcessMerge", 600, params)