How can I use MSSparkUtils in Python from PyCharm?

Question

How can I use MSSparkUtils in Python from PyCharm?

Victor Seifert 151

I have a script that works in Synapse by utilizing the mssparkutils to retrieve credentials via a LinkedService:

storage_token_source = mssparkutils.credentials.getConnectionStringOrCreds(linked_service_name_source)

However, if I add this code to my local repository, PyCharm throws an Error as mssparkutils is unknown - I couldn't find the package in PyPi, how can I add it to the repository as an import/dependency?

I package my code later on and deploy it as a whl file to the spark pools.

Best regards,
Victor

1 answer

Your answer

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

Hello @Victor Seifert ,

Thanks for the question and using MS Q&A platform.

You can download the mssparkutils package from below MVN repo and import the jar into your local project. With that, you should be able to compile the project successfully: https://mvnrepository.com/artifact/com.microsoft.azure.synapse/synapseutils

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Victor Seifert 151 Reputation points

2022-05-11T07:52:42.017+00:00
Correct me if I'm wrong, but isn't the .jar mainly for Java/Scala projects? I don't know of any way to import a .jar file in Python?

Of course I can add the jar to my spark config, but that won't solve the issues I have with my code-editor + tests not recognizing mssparkutils as a valid python reference (to the package) and throwing errors.

I solved it currently that my custom class expects two a parameters:

param a: flag whether the code is executed locally or in Synapse. (so any use of mssparkutils in local-execution-mode is omitted)

param b: mssparkutils as a reference to the package (which gets stored in a variable)

Once I execute it in Synapse, I pass the reference to the mssparkutils package to my custom package's class via the parameter which enables me to use it like a variable in the code.

This feels very ugly, but I don't see any other way not to have errors all over my code. If there is a better solution, please add to the answer, but the existing answer above is not actually a solution.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-05-11T08:11:01.667+00:00

Hello @Victor Seifert ,

The answer was provided by the internal product team.

We are reaching out to the internal team to get the more details on the better solution. I will be update you once I hear back from the team.
Spaccapanico Proietti Giada 1 Reputation point

2022-12-05T09:16:23.427+00:00

Hi Victor,
I would like to replicate your workaround. Can you send me a link to a repo in which you used your approach?
Thanks
Giada

Victor Seifert 151

It's all in company repos, but here is a shortened version, not pretty but should suffice:

def retrieve_storage_token(linked_service_name: str, account_name: str, container_name: str, mssparkutils=None, sas_token: str = None, account_key: str = None):  
    """The 'mssparkutils' parameter is the reference to the entire package within synapse - pass it like any other variable."""  
    if mssparkutils is not None:  
        # Executed within a Synapse Notebook  
        storage_sas_token = mssparkutils.credentials.getConnectionStringOrCreds(linked_service_name)  
    else:  
        # executed in "local" mode on your computer / anywhere outside of Synapse  
        if sas_token is not None:  
            spark.conf.set(  
                f"fs.azure.sas.{container_name}.{account_name}.dfs.core.windows.net",  
                sas_token,  
            )  
        elif account_key is not None:  
            spark.conf.set(f"fs.azure.account.key.{account_name}.dfs.core.windows.net", account_key)

I would call me function as follows:

if run_environment == "synapse":  
    retrieve_storage_token(...., mssparkutils=mssparkutils, sas_token=None, account_key=None)  
else:  
    retrieve_storage_token(...., mssparkutils=None, sas_token=myvar, account_key=myvar2)

The run_environment is a global variable that gets set to "local" at the beginning of the script - in Synapse I overwrite it in the Notebook and set it to "synapse".

Share via

How can I use MSSparkUtils in Python from PyCharm?

1 answer

Your answer