Workspace package ModuleNotFoundError after successfully provisioning to Spark pool

Daniel King 5

I have successfully uploaded a working python whl file as a package to Synapse and provisioned the package successfuly to my Spark Pool. However, when importing the package in a notebook running on my Spark Pool I'm receiving a ModuleNotFoundError. I have restarted the Spark Pool after provisioning and the custom package shows up in "!pip list" yet it cant seem to find the package when importing. Could this be an issue with the PATH the notebook kernel is looking in?

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Daniel King 5 Reputation points

2023-05-02T18:53:55.7866667+00:00

@Vahid Ghafarpour How exactly should I go about SSH into the driver node? I can't seem to find any resources online on how to do this on the latest version of Synapse.

Also as a side question, is this supposed to happen? Is this a common step for anyone that would like to import a custom python package after following Microsoft's documentation of steps to provision a package to Spark Pool?
PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-05-03T07:57:19.11+00:00

@Daniel King - Thanks for the question and using MS Q&A platform.

In order, to investigate further please do share the below details:

Share the library python whl file which you are trying to install?
And also, share the Apache Spark version ( 3.1, 3.2 or 3.3) which you are trying to use?

Looking forward for your response.
PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-05-03T08:14:54.01+00:00

@Vahid Ghafarpour - I would encourage you to stop posting answers using chatgpt.

This is against the code of conduct -Artificial Intelligence (AI) usage policy for Microsoft Q&A.

Even if you are using chatgpt to answer the question on MS Q&A platform, I would encourage you to spend sometime to validate the answer before posting as an answer.

If you provide the answers without validating, this may cause alot of confusion for the Original poster.
Daniel King 5 Reputation points

2023-05-03T13:06:18.88+00:00

@PRADEEPCHEEKATLA-MSFT Thank you for getting back to me.

I uploaded and provisioned the "google-auth" package as well as a simple custom package that has a single function just for testing purposes. Both have been tested and work on my local device.

The Spark Pool is version 3.3 and the Python version is 3.10.
PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-05-11T08:07:28.8166667+00:00

@Daniel King - We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

I have provided a detailed answer which has clear steps for which you are looking for. If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT

2 answers

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more
PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-05-04T04:32:16.92+00:00

@Daniel King - Azure Synapse Runtime for Apache Spark 3.3 - google-auth comes default with Apache Spark version 3.3 - you need not to install this package seperately.

As per the repro, I had used Apache Spark pool 3.3 and verifed that the google-auth package has been already installed.

When you say getting ModuleNotFoundError error message- could you please help with the command which you are running along with the screenshot of the code and the error message?

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Daniel King 5 Reputation points

2023-05-04T14:42:44.62+00:00

@PRADEEPCHEEKATLA-MSFT Google_auth and my custom package are not importing but they do show under the pkg-resources.working-set. Here is my custom package showing up from your same code:

Here is the same package showing up under "!pip list":

Directly importing does not work:

Using require to bring the package still does not work:

Finally I tried directly adding it to the system Python path and it still cannot import the package:

As for google-auth it appears that google-auth-oauthlib comes with Apache Spark 3.3 but google-auth does not.

My spark pool configuration (notice version 3.3 at the bottom):

This custom package has been tested locally and does work so the issue does not lie there.

PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-05-05T03:17:25.6766667+00:00

@Daniel King - Thanks for additional details.

Could you please help us understand more about the custom package?

What does the custom package include?
Is it possible to share custom package with us, so that we can test it from our end?

Looking forward for you response.

PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-05-10T08:02:59.5433333+00:00

@Daniel King - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment

Share via

Workspace package ModuleNotFoundError after successfully provisioning to Spark pool

2 answers