Spark pool requirements.txt install package from a repository manager other than PyPI

Question

Hello,

I am having some trouble installing packages in my Spark pool on Synapse.
My requirements.txt file looks something like this :
--extra-index-url <server_url>
<project_name>

The execution returns the following error : Invalid package name --extra-index-url
Why does the package installer interpret my option as a package name although I am following the official pip syntax ? Can you please help ?

Thank you for your help.

Answer 1

Hello @Anonymous ,

Welcome to the Microsoft Q&A platform.

Python packages can be installed from repositories like PyPI and Conda-Forge by providing an environment specification file.

Environment specification formats:

PIP requirements.txt

A requirements txt file (output from the pip freeze command) can be used to upgrade the environment. When a pool is updated, the packages listed in this file are downloaded from PyPI. The full dependencies are then cached and saved for later reuse of the pool.

The following snippet shows the format for the requirements file.

absl-py==0.7.0  
adal==1.2.1  
alabaster==0.7.10  

YML format (preview)

In addition, you can also provide an environment.yml file to update the pool environment. The packages listed in this file are downloaded from the default Conda channels, Conda-Forge, and PyPI. You can specify other channels or remove the default channels by using the configuration options.

This example specifies the channels and Conda/PyPI dependencies.

name: stats2  
channels:  
- defaults  
dependencies:  
- bokeh  
- numpy  
- pip:  
  - matplotlib  
  - koalas==1.7.0  

For more details, refer to Manage Python libraries for Apache Spark in Azure Synapse Analytics.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

@PRADEEPCHEEKATLA Is installing python packages from private repo is supported yet?