Packages in the Requirements file are not getting installed for Apache Spark Pool on Synapse

Swapnil kumar 26 Reputation points
2020-11-24T06:18:56.71+00:00

We have a requirements.txt file which we have uploaded in the packages section under the apache spark pool we have .
The requirements file looks something like this:
42080-image.png

However, on executing the sparkjob in the spark pool, we get the following error output which says it could not find the module :
42024-image.png

In order to confirm whether the packages mentioned in the Requirements.txt are installed , I rand the following code:

import pkg_resources  
for d in pkg_resources.working_set:  
     print(d)  

And the output confirmed that the packages in the file are not installed as they are not present in the pkg_resources

The thing to note here is thata the exact same spark job on the exact same pool with the exact same Requirements.txt ran perfectly fine a couple of weeks ago, but when I tried executing it yesterday I was facing these issues.

What we also noticed was that the packages section which used to appear in Manage -> Apache Spark Pool -> Properties of Synapse Studio is now missing. Not really sure if this is one of the symptoms of the problem we are facing, but i thought it would be good to bring it to your notice:
42091-image.png

Also, we did wait for about an hour of uploading the Requirements .txt file to execute the code but it still throws an error and it does not show up in the list of installed packages either.

Please guide me on the same as I can't figure out why a functioning thing suddenly stopped working . This is kind of urgent for us.
Thanks!

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,336 questions
{count} votes

2 answers

Sort by: Most helpful
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2020-12-09T01:39:26.743+00:00

    Hello @Swapnil kumar ,

    We just heard back and the fix is deployed now , can you please ret again and let us know if the is issue is resolved .
    Once again we sincerely apoloziges for the delay on this issue from our side .

    Thanks
    Himanshu


  2. Swapnil kumar 26 Reputation points
    2020-12-09T08:15:59.543+00:00

    I re-uploaded the requirements.txt file and now it is working fine. Thanks for the resolution!

    0 comments No comments