Package installation fails on Spark 2.4, succeeds in Spark 3.1.

Manuel Hernandez 1 Reputation point Microsoft Employee

I am trying to install a package in Azure Synapse Spark Pool 2.4 and I am getting the following error:

ERROR: Invalid requirement: 'idna==3.3 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff'

CondaValueError: pip returned an error.

The content of my environment.yml is the following:


  • pip:
  • idna==3.3 --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff
  • conda-forge

*The file is correctly indented

However the installation in Spark Pool 3.1 succeeds. According to the docs the version of pip should support passing these parameters. Is there something I am missing?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,346 questions
{count} votes