Parameterizing ADF Reference Names in ADF Pipeline JSON

Question

Hi Team,

We are trying to Parameterize the pipeline Json file with parameters and variables making it more generic in Azure Data Factory Pipelines without any hardcoding's. Except below reference name where Azure data factory objects are being used like linked service or integration runtime, we are able to parameterize the other hardcoding's. Can you help me to overcome this issue?

Below snip is reference for parameter which is giving an error

Thanks,

Manoj

Answer 1

In the Pipeline properties, add a new parameter:

"parameters": {

    "IntegrationRuntimeName": {

        "type": "string",

        "defaultValue": "shir-test-sample"  // this is optional

    }

}

In the connectVia section of your pipeline activity JSON, replace the hardcoded referenceName with the parameter reference.

"connectVia": {

    "referenceName": {

        "value": "@pipeline().parameters.IntegrationRuntimeName",

        "type": "Expression"

    },

    "type": "IntegrationRuntimeReference"

}

Here’s how the entire activity JSON might look after parameterization:

{

    "name": "MyCopyActivity",

    "type": "Copy",

    "dependsOn": [],

    "policy": {},

    "userProperties": [],

    "typeProperties": {

        "source": {

            "type": "BlobSource",

            "sourceRetryCount": 0

        },

        "sink": {

            "type": "SqlSink",

            "writeBatchSize": 10000

        }

    },

    "inputs": [

        {

            "referenceName": "InputDataset",

            "type": "DatasetReference"

        }

    ],

    "outputs": [

        {

            "referenceName": "OutputDataset",

            "type": "DatasetReference"

        }

    ],

    "connectVia": {

        "referenceName": {

            "value": "@pipeline().parameters.IntegrationRuntimeName",

            "type": "Expression"

        },

        "type": "IntegrationRuntimeReference"

    }

}

When you trigger this pipeline, you can pass the IntegrationRuntimeName parameter value:

{

    "IntegrationRuntimeName": "your-runtime-name"

}

Answer 2

Hi @ MADESHWARAN, MANOJ

Thanks for the question and using MS Q&A platform.

The error indicates that the ADF pipeline is unable to locate or load the necessary parameters defined in the NAStoNAS_DEV_V1_parameters resource. Here are some troubleshooting steps that might consider:

Verify JSON Syntax: Open the NAStoNAS_DEV_V1_parameters file and carefully inspect the JSON structure. Use a JSON validator to check for syntax errors. Ensure that all opening and closing brackets and quotes are correctly matched.

Check Resource Reference: Verify that the pipeline is referencing the NAStoNAS_DEV_V1_parameters resource correctly. Ensure that the resource name and path are accurate. Check if the resource exists in the ADF environment.

Inspect Resource Permissions: Verify that the user or service account running the pipeline has the necessary permissions to access the NAStoNAS_DEV_V1_parameters resource. Check for any access control lists (ACLs) or role-based access control (RBAC) restrictions.

Review Pipeline Logs: Examine the pipeline logs for more detailed error messages. Look for clues about the specific issue preventing the resource from loading.

Test in an Isolated Environment: If possible, try to isolate the issue by creating a simplified pipeline or testing the parameters in a different environment. This can help identify if the problem is specific to the pipeline or the parameters.

By systematically addressing these potential causes, you should be able to resolve the error and successfully load the NAStoNAS_DEV_V1_parameters resource in your ADF pipeline.

If the issue still persists, please do let us know. Glad to help.

I hope this information helps.