Need help on setting up Env variable in Azure Synapse Analytics Spark cluster

Question

Hi Team, I have dev, ppe and prod Azure Synapse analytics workspace. I will be deploying my jar for some computation. Now I need to insert the ENV variable for ppe, prod and dev enviorment. So that my jar running in spark cluster can refer env variable value and based on it, it can load dev, ppe or prod config. But I am unable to find out how to pass custom env variable for Synapse workspace.

Can you guys help me to know that?

Answer 1

Hello @Shikhil Kumar Gupta MSFT ,

Thanks for the question and using the MS Q&A platform.

You can read custom config values in a spark notebook as shown below.

Step1: Add your key-value pairs inside the config.txt file:

spark.executorEnv.environmentName ppe  

Step2: You can upload a config file to your spark pool.

Step3: Which you are then able to access inside your notebooks:

envName: str = spark.sparkContext.environment.get('environmentName', 'get')  

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators