Install Sentence-Transformers as a Workspace package on Azure Synapse Analytics

Haritha Thilakarathne 25 Reputation points MVP
2023-04-25T02:09:03.9666667+00:00

I'm trying to install sentence-transformers python package as a workspace package for Azure Synapse Analytics.(To be used on Apache Spark Pool) Tried installing it through magic command %pip install sentence-transformers through a notebook and it works. Following the steps for uploading the sentence-transformers-2.2.2.tar.gz to Workspace and configuration throws the following error mentioned in the screenshot. I tested with couple of other python packages and they work fine. Any workarounds? synapse_error

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,553 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 82,191 Reputation points Microsoft Employee
    2023-04-26T07:18:32.15+00:00

    Haritha Thilakarathne - Thanks for the question and using MS Q&A platform.

    PG Update(26/04/2023 2PM IST):
    Currently, we are working on fix on improve this error message and the documentation with the changes.

    The file .tar.gz format is only for R language packages (and not for python). To install python packages please use .yml or .whl file formats. 

    26/04/2023 11 AM IST:
    Looks like there is an issue with install whl file, which would leads to Job failed during run time with state=[dead] File not found. Please validate that file at the provided path exists and that you have the permissions required to access the file.
    As per the repro, I had tried to install the sentence-transformers-2.2.2.tar.gz whl file via workspace packages where I got the same error message as shown above. I will reach out to the internal product team to get more details on why exactly we are experiencing this error message. User's image

    Wheanwhile, you can install the sentence-transformers library by providing a requirements.txt file.
    Code snippet in requirement file: sentence-transformers==2.2.2 User's image

    We had successfully installed the sentence-transformers library using requirements.txt file. User's image

    Notebook shows you how to use an already trained Sentence Transformer model to embed sentences for another task. User's image

    Hope this helps. Do let us know if you any further queries.


0 additional answers

Sort by: Most helpful