Azure Synapse: How to use Spark job reference files?

Madeline Stager 31

I created a Spark job definition following the PySpark section of this tutorial.

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-job-definitions#create-an-apache-spark-job-definition-for-pyspark-python

I want to update additional python files in the reference files section of the job definition and use those files in the main python job file.
I tried a few different approaches and get errors that the classed I try to reference can't be found.

How can I use the reference python files in the main job definition?

PRADEEPCHEEKATLA 90,261 Reputation points

2020-12-17T10:59:33.883+00:00

Hello @Madeline Stager ,

Welcome to the Microsoft Q&A platform.

When you say "I tried a few different approaches and get errors that the classed I try to reference can't be found.", could you please share the different approaches which you have tried along with the error messages which you are experiencing?
PRADEEPCHEEKATLA 90,261 Reputation points

2020-12-21T04:55:14.83+00:00

Hello @Madeline Stager ,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.
Ed Freeman 6 Reputation points

2021-04-30T06:50:46.347+00:00

@Madeline Stager Did you find the solution to this?

@PRADEEPCHEEKATLA I've added a .py as a reference file in the same directory as the main definition file in the lake. It's not clear from the docs whether the reference files are pre-imported into the main Python script's context, or whether you have to explicitly import them. Anyway, I've tried both: I've referenced a class within the file without explicitly importing it, which didn't work. I tried importing the file using from <fileNameWithoutExtension> import <className> which didn't work either. It just errors saying "ModuleNotFoundError: No module named XXX" The latter should work in my opinion - this is how we reference another file when doing normal Python dev work.

Do you know how to do it successfully? Are there any examples of this?

Thanks, Ed
Madeline Stager 31 Reputation points

2021-04-30T18:33:52.597+00:00
@Ed Freeman I did find a solution. The reference files are add to the current working directory on the spark cluster when the job is run. I had to add the current working directory to the list of paths that python looks in for imported modules before the import statement.

sys.path.append(os.getcwd())
alexander tikhomirov 31 Reputation points

2022-04-07T13:55:31.64+00:00

but how to reference to file from another location, not from current, where main file located?

Share via

Azure Synapse: How to use Spark job reference files?

Your answer