Can we connect to multiple kusto clusters as source in an ADF pipeline in a parameterized way

Question

We have a scenario where our data needs to be kept in region until it can be aggregated and moved out. So we have multiple kusto clusters with different names but the same underlying DB's and tables. We want to set up ADF pipelines that can call each of the regions Kusto Clusters but call the same aggregation function or table in each region and union them together.

We certainly can hardcode these in the pipelines, however we may be adding more regions as time goes on and would like to just add the new Kusto Cluster name to some list of parameters. I see that we can parameterize DB's in a Cluster and loop through those. Here the cluster names are different but the DB and table names would be the same. Is this possible to loop though a list like this?

Answer 1

Hi @Eric Swanson ,

As per conversation with ADF product team, you can edit the JSON payload of you linked service to parameterize it as shown below:

{  
    "name": "AzureDataExplorer1",  
    "type": "Microsoft.DataFactory/factories/linkedservices",  
    "properties": {  
	    "parameters": {  
			"ADXCluster": {  
				"type": "String"  
			}  
		},  
        "annotations": [],  
        "type": "AzureDataExplorer",  
        "typeProperties": {  
            "endpoint": "https://@{linkedService().ADXCluster}.westus.kusto.windows.net",  
            "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx",  
            "servicePrincipalId": "ewafadsfaf",  
            "database": "abc",  
            "encryptedCredential": "ew0KICAiVmVyc2lvbiI6IxxxxxxxxxxODcxIg0KfQ=="  
        }  
    }  
}  

There is no UI support to assign values to ADX linked service parameters in Azure Data Explorer Command activity, but you can do it by manually edit the JSON code of your activity like below and it should work as expected.

{  
    "name": "pipeline4",  
    "properties": {  
        "activities": [  
            {  
                "name": "Azure Data Explorer Command1",  
                "type": "AzureDataExplorerCommand",  
                "dependsOn": [],  
                "policy": {  
                    "timeout": "7.00:00:00",  
                    "retry": 0,  
                    "retryIntervalInSeconds": 30,  
                    "secureOutput": false,  
                    "secureInput": false  
                },  
                "userProperties": [],  
                "typeProperties": {  
                    "command": "asasasa",  
                    "commandTimeout": "00:20:00"  
                },  
                "linkedServiceName": {  
                    "referenceName": "AzureDataExplorer1",  
                    "type": "LinkedServiceReference",  
                    "parameters": {  
                        "ADXCluster": "asasasas"  
                    }  
                }  
            }  
        ],  
        "annotations": [],  
        "lastPublishTime": "2021-08-31T06:15:47Z"  
    },  
    "type": "Microsoft.DataFactory/factories/pipelines"  
}  

I have provided a feedback to product team to include UI support for parameters in Azure Data Explorer Command activity and a work item has logged to track it. Hopefully this feature will be available in near feature.

Hope this helps. Do let us know if you have further query.

----------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Answer 2

Hi @Eric Swanson ,

Just wanted to update you that Product team has implemented a feature to parameterize Kusto linked service for multiple clusters.

Thank you for sharing your valuable feedback and helping us improve the Azure products.