Resolving Module Not Found Error in Azure Synapse Notebook

Question

  I've initiated a notebook in Azure Synapse workbook, attached an Apache Spark pool, and attempted to install a Python SDK using the command 'from XYZ import XYZ.client' in the first block of my notebook. However, I encounter an error stating 'Module "XYZ" not found.'

How can I resolve this issue? I have the Python wheel package for XYZ. Could someone provide guidance on the steps to follow in Azure Synapse?

Answer 1

@HideyoshiAkaio

Thank you for posting query in Microsoft Q&A Platform.

It seems like you’re trying to use a Python package that isn’t installed in your Apache Spark pool. Here are the steps you can follow to install the Python wheel package for XYZ:

Upload the Wheel File: First, you need to upload the .whl file of the XYZ package to a location accessible by the Spark pool. This could be a blob storage account associated with your Azure Synapse workspace.

You will get option to upload in different ways as below. First, enable Allow session level packages option, then under Workspace packages select the file you uploaded in workspace earlier and click on apply settings.

If you don't have any .whl files just create requirements.txt add required packages and upload here under Requirements files. After clicking on apply settings wait for some time to successfully apply settings. Then, go to notebook stop the current session and run notebook in new spark session. Make sure you run notebook in new session after adding packages.

Install the Package: Once the wheel file is uploaded, you can install it using the %pip install command in a notebook cell. The command should look something like this:

%pip install <path_to_wheel_file>

Replace <path_to_wheel_file> with the path where you uploaded the wheel file. If the file is in a blob storage, the path would be in the format wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<file_name>.whl.

Import the Module: After the package is installed, you should be able to import the module using the command you mentioned:

 

from XYZ import XYZ.client

Remember to restart the Spark session after installing the package to ensure that the newly installed package is recognized.

Please note that these steps will install the package for the current session only. If you want the package to be available in all sessions, you’ll need to install it on all nodes of the Spark pool. This can be done by adding a custom library to the Spark pool from the Azure Synapse Studio.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.