Using parameterized LinkedService in PySpark Notebook to access Gen1 DataLake

Question

Currently we are accessing Gen1 storage from PySpark notebook using Linked Service we have created in the workspace. The code snippet is as below:

sc._jsc.hadoopConfiguration().set("spark.storage.synapse.linkedServiceName", "<Gen1LinkedServiceName>")
sc._jsc.hadoopConfiguration().set("fs.adl.oauth2.access.token.provider", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProviderGen1")

This works fine but since we have different environments, we cannot use this as is. One way to resolve this is by parameterizing the LinkedService. However, we don't find any documentation on how to pass the parameters to that LinkedService from PySpark Notebook. Is there a way to do this?

Answer 1

@Soyab - Thanks for the question and using MS Q&A platform.

You can simply parameterize linked_service_name varaibale as the toggle parameter cell to pass the parameters to that LinkedService from PySpark Notebook

To parameterize your notebook, select the ellipses (...) to access the more commands at the cell toolbar. Then select Toggle parameter cell to designate the cell as the parameters cell.

Azure Data Factory/Synapse looks for the parameters cell and uses the values as defaults for the parameters passed in at execution time. The execution engine will add a new cell beneath the parameters cell with input parameters to overwrite the default values.

For more details, refer to Transform data by running a Synapse notebook - Passing parameters.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.