I'm developing a PySpark application using the typical Python structure, where we can import modules from other parts of the code into a script, such as:
from another_module.another_filename import MyFunction
These are functions that are either related to our existing architecture or to business, so we can't get rid of them.
When we have full control of our environment, is as easy as adding the code folder to the PYTHONPATH, but I suspect it cannot be done with the current Synapse configuration.
Right now, the Spark Job Definition template allows you to specify reference files. However, it doesn't matter were you store your files in the DataLake because they all the files end at the same level without hiearchy preservation in a specific folder of each Spark worker, thus making impossible to follow the aforementioned line of code.
Should there by any way to overcome, please let me know.