Install Python pycryptodome libraries for Apache Spark in Azure Synapse

Question

Hi All,

I am tying to install pycryptodome to azure synapse notebook. PFB details.

scenario - I have created a notebook and Apache spark pool in azure synapse. I used the Below command to list the packages installed on pool. I don't see my required packages in the list. So I tried to install it using requirement.txt file and requirement.yml file in package section of Apache spark pool.

steps performed-

pip list : command to see the packages already installed.

created below file and uploaded in package section of Apache Spark pool.

requirement.txt:

pycryptodome==3.16.0

requirement.yml code:

name: pycrypto_lib

channels:

• defaults

dependencies:

• pip:
  • pycryptodome

error= PFA screenshot

Error_Screenshot.png

please share your suggestion. Thanks!

Answer 1

Hello @sachin gupta

Thanks for the question and using MS Q&A platform.

As per the repro from my end - I'm able to successfully install the pycryptodome==3.16.0 using the requirement.txt file method.

Note: I had tested on Spark Runtime 3.1 version. In case, if you spark runtime is different please do let me know I can re-verfiy with the spark runtime version which you are using.

To verify if the correct versions of the correct libraries are installed from PyPI, run the following code:

import pkg_resources
for d in pkg_resources.working_set:
     print(d)

Successfully installed & verfied the package installtion named: pycryptodome==3.16.0

For more details, refer to Manage session-scoped packages.

Hope this helps. Do let us know if you any further queries.

---

Please don’t forget to Accept Answer wherever the information provided helps you, this can be beneficial to other community members.