Issue with Parameterizing Azure ML Linked Service in ADF

Question

Issue with Parameterizing Azure ML Linked Service in ADF

Sama Praveen Kumar Reddy 0

Description:
I am facing an issue while migrating an Azure Data Factory (ADF) pipeline to a higher environment using a CI/CD pipeline.

In ADF, I have an Azure Machine Learning (Azure ML) activity that connects using an Azure ML linked service.

Currently, the values in the ML linked service are hardcoded.

While migrating the ADF pipeline using CI/CD, I am unable to create parameters for the Azure ML linked service to pass the values dynamically.

I want to configure the linked service to use system-assigned managed identity or service principal instead of hardcoded values.

Issue:

How can I parameterize the Azure ML linked service in ADF to support dynamic connections during CI/CD deployment?

What is the correct approach to passing authentication credentials dynamically using system-assigned managed identity or service principal in CI/CD?

Expected Outcome:

I need guidance on dynamically configuring the Azure ML linked service in ADF so that it works seamlessly across different environments during CI/CD deployment.

Any insights or best practices would be appreciated.

Smaran Thoomu 21,620 Reputation points Microsoft External Staff

2025-02-14T11:33:12.8866667+00:00

@Sama Praveen Kumar Reddy Just checking in to see if the below answer provided by @Sina Salam helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

Smaran Thoomu 21,620 Reputation points Microsoft External Staff

2025-02-14T11:33:12.8866667+00:00

@Sama Praveen Kumar Reddy Just checking in to see if the below answer provided by @Sina Salam helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Sina Salam 18,876

Hello Sama Praveen Kumar Reddy,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having issue with Parameterizing Azure ML Linked Service in ADF.

Regarding your explanations, perform the steps to dynamically configure the Azure ML linked service in ADF and ensure seamless CI/CD deployment across different environments:

Edit the JSON definition of the linked service to include parameters for servicePrincipalId, servicePrincipalKey, and tenant. For an example:

     {
       "name": "AzureMLLinkedService",
       "properties": {
         "type": "AzureML",
         "typeProperties": {
           "servicePrincipalId": "@{linkedService().parameters.servicePrincipalId}",
           "servicePrincipalKey": "@{linkedService().parameters.servicePrincipalKey}",
           "tenant": "@{linkedService().parameters.tenant}"
         }
       }
     }

https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services

Use Azure Key Vault to:
1. Store the service principal credentials in Azure Key Vault.
2. Reference the secrets in the linked service using parameters.
Enable System-Assigned Managed Identity to:
1. Enable system-assigned managed identity for your ADF instance.
2. Assign the necessary roles to the managed identity in Azure ML and other resources.

Create a Service Principal:

Create a service principal in Azure Active Directory.
Assign the necessary roles to the service principal.
Store the service principal credentials in Azure Key Vault.

CI/CD Pipeline Configuration: Use ARM templates to define your ADF pipeline and linked services. Parameterize the ARM templates to pass environment-specific values during deployment. Use Azure DevOps to deploy the ARM templates and pass the parameters dynamically.
Best Practices are to: Use global parameters in ADF for common values. Ensure appropriate RBAC roles are assigned. Test the CI/CD pipeline in a lower environment before production deployment.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Sama Praveen Kumar Reddy 0 Reputation points

2025-02-13T13:43:34.0233333+00:00

Sina, Thanks for the clarification.

I edited the JSON definition of the linked service to create all parameter values, including SubscriptionId, ResourceGroupName, MLWorkspaceName, Tenant, ServicePrincipalID, KeyVaultName, and SecretName. After making these changes, I was able to test the connection successfully when selecting the Machine Learning pipeline ID type as Pipeline ID.

However, when I try to select the Machine Learning pipeline name from the dropdown, no results are found. Interestingly, if I create the linked service in the same way but provide hardcoded values, I am able to fetch the ML pipeline list in drop down and select my pipeline from the dropdown.

Could you please confirm if this behavior is expected or if there is a workaround for dynamically retrieving the pipeline list?

Thank
Praveen
Sama Praveen Kumar Reddy 0 Reputation points

2025-02-13T14:09:12.34+00:00

Thanks Sina, for the clarification.

I edited the JSON definition of the linked service to create all parameter values, including SubscriptionId, ResourceGroupName, MLWorkspaceName, Tenant, ServicePrincipalID, KeyVaultName, and SecretName. After making these changes, I was able to test the connection successfully when selecting the Machine Learning pipeline ID type as Pipeline ID.

However, when I try to select the Machine Learning pipeline name from the dropdown, no results are found. Interestingly, if I create the linked service in the same way but provide hardcoded values, I am able to fetch the ML pipeline list in drop down and select my pipeline from the dropdown.

Could you please confirm if this behavior is expected or if there is a workaround for dynamically retrieving the pipeline list?

Thanks
Praveen
Sina Salam 18,876 Reputation points

2025-02-13T15:14:55.54+00:00
Hello Sama Praveen Kumar Reddy,

Thank you for your feedback.

Initially, I provided a comprehensive solution for parameterizing the Azure ML linked service in ADF, including steps to use Azure Key Vault, system-assigned managed identity, and service principal.

However, the inability to dynamically retrieve the Machine Learning pipeline list in the dropdown when using parameterized values. This behavior was not anticipated in the original problem statement. That's a new issue.

To dynamically retrieving the Machine Learning pipeline list do the followings:

Ensure that the parameters for SubscriptionId, ResourceGroupName, MLWorkspaceName, Tenant, ServicePrincipalID, KeyVaultName, and SecretName are correctly defined and referenced in the linked service JSON.

Make sure the service principal or managed identity has the necessary permissions to list the Machine Learning pipelines. This includes roles such as Reader or Contributor on the Azure ML workspace.

Also, ensure that the secrets stored in Azure Key Vault are correctly referenced in the linked service. This can help avoid issues with parameterized values not being resolved correctly.

You can enable debugging and use the Azure Data Factory monitoring to check for any errors or warnings when attempting to retrieve the pipeline list. This can provide insights into what might be going wrong.

Refer to the Azure Data Factory documentation on parameterizing linked services for more details: https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services

NOTE:

If the issue persists, consider using a script or custom activity to fetch the list of Machine Learning pipelines and pass it to the ADF pipeline as a parameter. This can be done using Azure Functions or Azure Logic Apps as a workaround.

Goodluck
Smaran Thoomu 21,620 Reputation points Microsoft External Staff

2025-02-17T12:18:05.5466667+00:00

@Sama Praveen Kumar Reddy Just checking in to see if the above answer provided by @Sina Salam helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Issue with Parameterizing Azure ML Linked Service in ADF

1 answer

Your answer