Can we customize the index url to fetch Python package from a protected PyPi source
Recently Databricks provided in public preview the possibility to add library in the compute policy. We will need to install a library located in a protected PyPi repo from Azure DevOps: pkgs.dev.azure.com/{further_path}/pypi/packages.
We used to have the package installation in init script, where we were able to install the library by providing a token in the url by taking it from secret scope from Azure keyvault: https://{secret_from_kv}@pkgs.dev.azure.com/{further_path}. I have provided an example in the attachment.
Is there a way to provide a variable inside the index_url of the library in the compute policy? Maybe providing it either as a Spark environment variable or a reference to a secret from secret scope?
Or maybe a way to enable Databricks workspace to be able to fetch the package from Azure Artificats without providing the token in the path? Maybe setting the permissions on the Artifacts side? To allow Databricks public subnet or the service to install the library?
Best regards,
Aleksandra