ADF and Azure ML Integration

Question

Hello,

In Azure Data Factory (ADF), we have Development (DEV), Staging (Stage), and Production (PROD) workspaces. Similarly, Azure Machine Learning (Azure ML) also has corresponding DEV, Stage, and PROD workspaces.

In ADF DEV, I have an Azure Machine Learning (Azure ML) activity that connects using an Azure ML linked service. Currently, the ML linked service contains hardcoded values for subscriptionId, resourceGroupName, and WorkspaceName. With this setup, I am able to connect to the Machine Learning pipeline and successfully execute it through ADF.

However, when migrating the Machine Learning linked service to a higher environment (Stage/PROD), I am encountering issues due to the hardcoded values.

• In Stage, it attempts to fetch Stage ML details.
• In PROD, it fetches PROD ML details.

To resolve this, I am passing the required values in a parameterized JSON format. The connections test successfully, but unfortunately, I am unable to fetch the Machine Learning pipeline name details.

Interestingly, if I configure the linked service with hardcoded values, I am able to fetch the ML pipeline list and execute the pipeline through ADF without issues.

below json request
{

"properties": {

    "type": "AzureMLService",

    "parameters": {

        "subscriptionId": {

            "type": "String"

        },

        "resourceGroupName": {

            "type": "String"

        },

        "mlWorkspaceName": {

            "type": "String"

        }

    },

    "annotations": [],

    "typeProperties": {

        "subscriptionId": "@{linkedService().parameters.subscriptionId}",

        "resourceGroupName": "@{linkedService().parameters.resourceGroupName}",

        "WorkspaceName": "@{linkedService().parameters.WorkspaceName}",

        "authentication": "MSI"

    }

}

}

Could you please provide insights or recommendations on resolving this issue?

Thank you.

Best regards,
praveen

Answer 1

Thank you for your response and valuable insights.

We are currently using a classic CI/CD pipeline to create parameters and pass the corresponding values for Stage and PROD. However, we are facing an issue where the ML pipeline list is not being fetched. Given this limitation, creating parameters for the linked service without successfully retrieving the ML pipelines seems ineffective.

Could you please guide us on the best approach to passing the corresponding values for the linked service in Stage and PROD while using a classic CI/CD pipeline? Any recommendations or best practices would be greatly appreciated.

Looking forward to your guidance.

Answer 2

Hi @Sama Praveen Kumar Reddy

Thank you for your follow-up and for providing additional context regarding your CI/CD pipeline setup.

You're absolutely right. If you can't browse and select the ML pipeline in your DEV environment because of the parameterization, then the pipeline name won't be saved in your ADF definition, and therefore it won't be executed in Stage/Prod.

To address this effectively, here are some recommendations for managing your parameterized linked services in the classic CI/CD pipeline while ensuring successful ML pipeline execution:

Maintain separate configuration files for each environment (DEV, Stage, PROD) that contain the necessary values for subscriptionId, resourceGroupName, and mlWorkspaceName. During the CI/CD pipeline execution, ensure the correct configuration file is used to populate these parameters before deploying to the respective environment.

Store sensitive information such as subscription IDs and workspace names in Azure Key Vault. Use managed identities to access these values securely during runtime, ensuring that your deployments are both dynamic and secure.

Store your environment-specific values in Azure DevOps variable groups. You can create different variable groups for DEV, Stage, and PROD. During the deployment process, use these variable groups to dynamically set the values in your linked service parameters.

Define your ADF linked services in an ARM template and use parameter files for each environment. This will allow you to specify different values for each environment and deploy them using your CI/CD pipeline.

Implement logging and monitoring in your CI/CD pipeline to capture any errors or issues during deployment. This will help you troubleshoot problems more effectively.

Reference documentations:

• Parameterize linked services in Azure Data Factory and Azure Synapse Analytics
• Troubleshoot CI-CD, Azure DevOps, and GitHub issues in Azure Data Factory and Synapse Analytics

Hope this helps. Do let us know if you have any further queries.

Answer 3

Hello Sama Praveen Kumar Reddy,

Thank you for reaching out on the Microsoft Q&A.

When you parameterize the Azure ML linked service in ADF, fetching the ML pipeline list doesn't work like it does with hardcoded values. This happens because ADF needs static values to fetch the pipelines at design time.

To fix this:

1. For development, use hardcoded values to fetch the pipeline list.
2. Once you're ready to deploy to Stage or PROD, parameterize the linked service with the correct values for subscriptionId, resourceGroupName, and mlWorkspaceName.

This way, the connection will work across environments, but you won't be able to fetch pipelines at design time with parameterized values. For more details, check out ADF Parameterization and Execute ML Pipelines.

I hope this helps! If you have any further questions, feel free to ask.

If the information is useful, please accept the answer and upvote it to assist other community members.