Pyspark HDInsight DataFactory Eviroment Variable

ALBERTO JUNIOR 1 Reputation point
2021-09-08T21:57:22.217+00:00

I'm faced a problem with Pyspark, datafactory and HdInsight

I create a HDInsight with 2 master and 2 slaves.

I created environment variables in all server like

sudo echo 'TEST=server' >> /etc/environment

After that, in all server I opened sever and executed in terminal

  • pyspark
  • from os import environ as env
  • test = env.get("test")
  • print(test)
    The code will print - test

But when I use datafactory and execute spark-submit I can access the value of my variable

.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,525 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
201 questions
{count} votes