Pyspark HDInsight DataFactory Eviroment Variable
ALBERTO JUNIOR
1
Reputation point
I'm faced a problem with Pyspark, datafactory and HdInsight
I create a HDInsight with 2 master and 2 slaves.
I created environment variables in all server like
sudo echo 'TEST=server' >> /etc/environment
After that, in all server I opened sever and executed in terminal
- pyspark
- from os import environ as env
- test = env.get("test")
- print(test)
The code will print - test
But when I use datafactory and execute spark-submit I can access the value of my variable
Sign in to answer