Newest update to apache spark pool breaks geopandas install using conda

Banaszczyk, Kamil 22 Reputation points
2022-11-04T08:14:27.047+00:00

Hello,

During recent deployment of our application we noticed that we cannot deploy our library which in fact blocks our release.

I believe root cause is recent update to apache spark pool 2.4 which is only mentioned briefly as a note under this page:
https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-24-runtime

Currently when i'm uploading environment.yml file to apache spark job configuration that looks like this:

name: root
channels:

  • anaconda
    dependencies:
  • geopandas=0.9.0

At the end of LibraryManagement jobs I receive this kind of error:

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your system:

  • feature:/linux-64::__glibc==2.27=0
  • feature:|@/linux-64::__glibc==2.27=0

Your installed version is: 2.27

Before that update everything was working fine. Do you know if there is any workaround that we can apply to not break our CD process ?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,558 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 82,671 Reputation points Microsoft Employee
    2022-11-28T06:40:56.4+00:00

    Hello @Banaszczyk, Kamil ,

    Here is the response from the internal team:

    Yes last month, as part of upgrading conda package manager, dependent libgcc libraries were upgraded as well. I can see this upgrade is causing cx package geopandas to conflict. This is an expected impact of the conda upgrade. Make sure to use this env.yml file instead. I tried this on my test workspace and it works.

    name: root  
    channels:  
    - anaconda  
    dependencies:  
    - pip:  
      - geopandas==0.9.0  
    

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful