Can we customize the index url to fetch Python package from a protected PyPi source

Sypula, Aleksandra 0

Recently Databricks provided in public preview the possibility to add library in the compute policy. We will need to install a library located in a protected PyPi repo from Azure DevOps: pkgs.dev.azure.com/{further_path}/pypi/packages.

We used to have the package installation in init script, where we were able to install the library by providing a token in the url by taking it from secret scope from Azure keyvault: https://{secret_from_kv}@pkgs.dev.azure.com/{further_path}. I have provided an example in the attachment.

Is there a way to provide a variable inside the index_url of the library in the compute policy? Maybe providing it either as a Spark environment variable or a reference to a secret from secret scope?

Or maybe a way to enable Databricks workspace to be able to fetch the package from Azure Artificats without providing the token in the path? Maybe setting the permissions on the Artifacts side? To allow Databricks public subnet or the service to install the library?

Best regards,

Aleksandra

PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-03-28T05:25:51.02+00:00
@Sypula, Aleksandra -

@Sypula, Aleksandra - Thanks for the question and using MS Q&A platform.

Yes, you can provide a variable inside the index_url of the library in the compute policy. You can use the Databricks Secrets API to retrieve the secret value from Azure Key Vault and then use it in the index_url.

Here is an example:

First, you need to create a secret scope in Databricks and add the Azure Key Vault as a backing key store. You can follow the instructions in this document: Azure Databricks - Azure Key Vault-backed scopes

Once you have created the secret scope, you can add a secret to it. In your case, you can add the token for the protected PyPi repo as a secret.

In your compute policy, you can reference the secret using the Databricks Secrets API. Here is an example:

{ "pypi": { "index_url": "https://{secret:my-secret-scope:my-secret}@pkgs.dev.azure.com/{further_path}/pypi/packages" } }

In this example, my-secret-scope is the name of the secret scope you created in step 1, and my-secret is the name of the secret you added in step 2.

You can also use environment variables to store the secret value and reference it in the index_url. Here is an example:

{ "pypi": { "index_url": "https://${env:MY_SECRET}@pkgs.dev.azure.com/{further_path}/pypi/packages" } }

In this example, MY_SECRET is the name of the environment variable that contains the secret value.

Regarding your second question, you can configure Azure Artifacts to allow Databricks to access the package without providing the token in the path. You can follow the instructions in this document: Get started with NuGet packages in Azure Artifacts. Once you have configured Azure Artifacts, you can reference the package in your compute policy using the standard index_url for Azure Artifacts.

For more details, refer to Azure Databricks - Create and manage compute policies.

Hope this helps. Do let us know if you any further queries.
Sypula, Aleksandra 0 Reputation points

2024-03-29T08:59:36.6433333+00:00

Thank you very much for the answer. One more question as in the documentation about adding a library, there is only information about adding the library manually via the UI: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/clusters/policies#policy-def and when providing in the index_url the value with ${secret}@ it throws error about incorrect characters.
Also on the Databricks API documentation, https://docs.databricks.com/api/workspace/clusterpolicies/create in the library, there are only 2 fields mentioned: package and repo:

Could you please advise how I could define the policy with the library in the UI? Or maybe where to define the pypi in the compute policy? When I provided the "pypi" in the json definition of the compute policy it warned that the value is not recognized
Sypula, Aleksandra 0 Reputation points

2024-03-29T09:06:11.0666667+00:00

Hi @PRADEEPCHEEKATLA-MSFT thank you very much for your answer. One more question to the solution, when I try to provide the index_url with the ${secret}@ via the UI for adding a library to the compute policy it throws an error about invalid characters. Also in the docummentation: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/clusters/policies#policy-def it is not mentioned how I could define the policy as a json file. When tired to add the "pypi" field to the json policy definition it warned that given name is not recognized.
Also in the Darabricks API documentation, in the library definitoin, there is only repo and package fields mentioned: https://docs.databricks.com/api/workspace/clusterpolicies/create

Could you please advise how I can add the index_url with the secret via UI or how to define the compute policy definition?
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-01T06:41:59.13+00:00

@Sypula, Aleksandra - We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
Sypula, Aleksandra 0 Reputation points

2024-04-08T08:40:43.7433333+00:00

Hi @PRADEEPCHEEKATLA-MSFT could you please let me know if there are any updates?
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-08T10:20:07.5766667+00:00

@Sypula, Aleksandra - I am assuming you have tried it and it did not work. If that is true, you would have to use an init_script option available in compute policy and install the library, rather than using the "add libraries to a compute policy". Hope this helps. Do let us know if you any further queries.
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-12T06:22:22.8666667+00:00

@Sypula, Aleksandra - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.