Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,403 questions
Changing Synapse Notebook variable values during deployment in Azure DevOps
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 41%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
ADITYA SINGH
0
Reputation points
How can I change the variable values in a Synapse Notebook, such as storage account name, container name, and file location, during deployment from dev to prod in Azure DevOps? The notebook is already being used by other notebooks in my dev environment.
{count} votes
-
Vinodh247-1375 12,426 Reputation points2024-05-29T05:16:57.1233333+00:00 by making your synapse notebook parameterized, you can define parameters for the variables you have listed like below
storage_account_name = dbutils.widgets.get("storage_account_name") container_name = dbutils.widgets.get("container_name") file_location = dbutils.widgets.get("file_location") storage_account_url = f"abfss://{container_name}@{storage_account_name}.dfs.core.windows.net" data_path = f"{storage_account_url}/{file_location}" # Your existing code using data_pathand then define variables in your devops pipelines, you can set the variables at jobs in your pipeline.
then use your synapse pipeline to run your notebooks with the required parameters. When you trigger the deployment, the pipeline will use the variables defined and pass them as parameters to your synapse notebook.
-
Vinodh247-1375 12,426 Reputation points
2024-05-29T05:18:17.7133333+00:00 another approach is to use the custom parameters template as detailed below
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga 5,270 Reputation points Microsoft Vendor2024-05-29T09:07:42.47+00:00 Hi ADITYA SINGH
Welcome to Microsoft Q&A platform and thanks for posting your question here.To manage Synapse Notebook variable values during deployment in Azure DevOps, especially when the notebook is being used by other notebooks in your dev environment, you can follow these best practices:
Parameterization:
The first step is to convert all environment-specific values in your notebook to parameters. This includes storage account names, container names, and file locations. You can use the%%synapsemagic command to define parameters at the beginning of your notebook. By parameterizing your notebook, you can easily change the values based on the deployment environment.- ARM Template Configuration:
The next step is to use ARM templates to define the infrastructure and configurations for your Synapse workspace. You should include parameters in the ARM template that correspond to the ones in your notebook. This allows you to deploy the same infrastructure and configurations across different environments, while still being able to customize the values based on the environment. - Azure DevOps Pipeline:
In your Azure DevOps pipeline, you should use variable groups or pipeline variables to store environment-specific values. During the deployment phase, you can pass these values to the ARM template parameters. This ensures that the correct values are used during deployment, based on the environment. - Linked Services:
If your notebook is referencing linked services, you should ensure that these services are also parameterized in the ARM template. This allows you to dynamically connect to different data stores based on the deployment environment. By parameterizing the linked services, you can easily change the connection details based on the environment. - Dependency Management:
Since your notebook is used by other notebooks, you should ensure that the dependent notebooks also have their parameters updated to reflect the changes. This might involve updating the parameters in the linked notebooks themselves or ensuring they are set to inherit the updated values. By updating the dependent notebooks, you can ensure that they continue to work correctly after the parameter changes. - Testing:
Before deploying to production, it's important to test the deployment in a staging environment. This helps to catch any issues with parameter substitution or linked services. By testing in a staging environment, you can ensure that the deployment works correctly before deploying to production.
Hope this helps. Do let us know if you any further queries.
- ARM Template Configuration:
-
Harishga 5,270 Reputation points Microsoft Vendor
2024-05-30T07:28:43.9533333+00:00 @ADITYA SINGH
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment