Share via

Azure Synapse Analytics - can I downgrade Python libraries?

AlexW.admin 21 Reputation points
2021-06-23T11:08:17.443+00:00

Hi

I'm wondering if it's possible to upgrade/downgrade Python libraries for apache spark application if they're part of the core set: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-24-runtime

When I try to change any of these libraries using a requirements.txt, I get the following error, which leads me to believe it may not be possible which is annoying:

Error code 1
LIVY_JOB_STATE_DEAD

Message
[plugins.ultsynapse.systemreservedpool-librarymanagement.16 CCID:<d3b34775-25fb-46dc-bc88-907c81fb1ab5>] [Monitoring] Livy Endpoint=[https://hubservice.uksouth.azuresynapse.net:8001/api/v1.0/publish/18978b88-bb52-4e21-b7fe-997ef98d9a6f]. Livy Id=[0] Job failed during run time with state=[dead].

Edit: Seems to be working now as I believe I made a typo in the requirements text (= instead of ==). Sorry, idk if this question will still be useful to other people or not.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,379 questions
Developer technologies | .NET | Other
0 comments
Accepted answer
  1. Ronen Ariely 15,206 Reputation points
    2021-06-23T11:33:11.093+00:00

    Good day,

    can I downgrade Python libraries?

    You can manage Python libraries for Apache Spark in Azure Synapse Analytics. In fact extra Python and custom-built packages can be added at the Spark pool and session level.

    You have a full tutorial in this doc:
    https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-manage-python-packages

    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.