Package installation fails on Spark 2.4, succeeds in Spark 3.1.

Manuel Hernandez 1 Microsoft Employee

I am trying to install a package in Azure Synapse Spark Pool 2.4 and I am getting the following error:

ERROR: Invalid requirement: 'idna==3.3 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff'

CondaValueError: pip returned an error.

The content of my environment.yml is the following:

dependencies:

pip:

idna==3.3 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff
channels:

conda-forge

*The file is correctly indented

However the installation in Spark Pool 3.1 succeeds. According to the docs the version of pip should support passing these parameters. Is there something I am missing?

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2022-01-20T08:07:59.77+00:00
Hello @Manuel Hernandez ,

Thanks for the question and using MS Q&A platform.

Please try to pin your idna version in your environment.yml to 2.05 for Spark 2.4 runtime and see if that helps. Pip usually complains above error if any of your top-level dependencies/sub-dependencies are incompatible. Seems like idna version idna==3.3 is compatible with pip=21.1.1 (Spark 3.1) but not with pip=20.2.4 (Spark 2.4)

dependencies:

pip:

idna==2.05 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff
channels:

conda-forge

As a best practice, you can adopt a tool like pip-tools where you will be able to pin your top-level dependencies in a requirements.in file and run a pip-compile command to generate the requirements.txt file. This way there will be an explicit delineation between the top-level dependencies and the sub-dependencies. Also, the tool will resolve the sub-dependency conflicts for you.

Hope this info helps. Do let us know how it goes. If you are still blocked, would it be possible to tell us what is the package you are trying to install? Thank would help us to reproduce the issue on our end.
Manuel Hernandez 1 Reputation point Microsoft Employee

2022-01-20T18:37:49.227+00:00

Thanks for your reply @KranthiPakala-MSFT
I tried installing the same dependency without the hash check and the installation went through.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

requests 2.24.0 requires idna<3,>=2.5, but you'll have idna 3.3 which is incompatible.
mlflow-skinny 1.20.2 requires importlib-metadata!=4.7.0,>=3.7.0, but you'll have importlib-metadata 1.7.0 which is incompatible.
INFO - Fetching package metadata ...........
Solving package specifications: .
Collecting idna==3.3
Downloading idna-3.3-py3-none-any.whl (61 kB)
Installing collected packages: idna
Attempting uninstall: idna
Found existing installation: idna 2.10
Uninstalling idna-2.10:
Successfully uninstalled idna-2.10
Successfully installed idna-3.3

I was looking online and it seems that flags may not be available for conda < 4.4.x
https://github.com/conda/conda/pull/3969

Seems that spark 2.4 is using conda=4.3.21. Do you think this would be related?
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2022-02-01T00:56:27.73+00:00

Hello @Manuel Hernandez ,

Thanks for sharing additional findings. As you confirmed that installation went through fine without HASH then I suspect there could be something with Conda version. As you are internal to MSFT could you please reach out to me over teams so that we can dig more on it. Or we could also escalate to relevant product team, or we may have to file a support ticket for deeper investigation. Please let us know how you would want to proceed.

Thank you
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2022-02-03T07:06:20.443+00:00

Hi there,

As I haven't heard back from you, following up to see if you still need assistance on this query. In case if you have already found a solution, please do consider sharing it here with community, as it would be helpful for others reading this thread. If you still need assistance, please feel free reach out to me as mentioned in my previous comment.