Package installation fails on Spark 2.4, succeeds in Spark 3.1.

Manuel Hernandez 1 Reputation point Microsoft Employee
2022-01-18T17:52:21.78+00:00

I am trying to install a package in Azure Synapse Spark Pool 2.4 and I am getting the following error:

ERROR: Invalid requirement: 'idna==3.3 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff'

CondaValueError: pip returned an error.

The content of my environment.yml is the following:

dependencies:

  • pip:
  • idna==3.3 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff
    channels:
  • conda-forge

*The file is correctly indented

However the installation in Spark Pool 3.1 succeeds. According to the docs the version of pip should support passing these parameters. Is there something I am missing?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,346 questions
{count} votes