question

EricSwanson-8494 avatar image
0 Votes"
EricSwanson-8494 asked KranthiPakala-MSFT answered

Can we connect to multiple kusto clusters as source in an ADF pipeline in a parameterized way

We have a scenario where our data needs to be kept in region until it can be aggregated and moved out. So we have multiple kusto clusters with different names but the same underlying DB's and tables. We want to set up ADF pipelines that can call each of the regions Kusto Clusters but call the same aggregation function or table in each region and union them together.

We certainly can hardcode these in the pipelines, however we may be adding more regions as time goes on and would like to just add the new Kusto Cluster name to some list of parameters. I see that we can parameterize DB's in a Cluster and loop through those. Here the cluster names are different but the DB and table names would be the same. Is this possible to loop though a list like this?

azure-data-factory
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ericswanson-8494,

Welcome to Microsoft Q&A forum and thanks for posting your query.

Could you please confirm if you are referring to Azure Data Explorer linked service or Kusto Linked service?

126176-image.png

Please confirm so that we can better assist on this requirement.

0 Votes 0 ·
image.png (61.4 KiB)

Hi @ericswanson-8494,

Just checking to if you have got a chance to see my previous comment, if so could you please confirm the details?

0 Votes 0 ·

Hi @KranthiPakala-MSFT we have a bunch of kusto clusters (azure data explorer) that we are trying to connect to and call all the same underlying db and function name to copy the same schema data from multiple regions and combine into one aggregated local region (also an azure data explorer instance). Thanks.

0 Votes 0 ·

Hi @ericswanson-8494,

Thanks for your response and clarifying the ask. I did tried to edit Azure Data Explorer linked service to include parameters for the cluster name but it is not reflecting in the UI when tried to use the linked service in a data set.

Hence I can confirm that parameterization of cluster name for ADX linked service is currently not supported.
I am escalating this to ADF product team and will get back to you as soon as I have an update from the team.

Thank you for your patience.

0 Votes 0 ·
KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @ericswanson-8494,

As per conversation with ADF product team, you can edit the JSON payload of you linked service to parameterize it as shown below:

 {
     "name": "AzureDataExplorer1",
     "type": "Microsoft.DataFactory/factories/linkedservices",
     "properties": {
         "parameters": {
             "ADXCluster": {
                 "type": "String"
             }
         },
         "annotations": [],
         "type": "AzureDataExplorer",
         "typeProperties": {
             "endpoint": "https://@{linkedService().ADXCluster}.westus.kusto.windows.net",
             "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxx",
             "servicePrincipalId": "ewafadsfaf",
             "database": "abc",
             "encryptedCredential": "ew0KICAiVmVyc2lvbiI6IxxxxxxxxxxODcxIg0KfQ=="
         }
     }
 }


There is no UI support to assign values to ADX linked service parameters in Azure Data Explorer Command activity, but you can do it by manually edit the JSON code of your activity like below and it should work as expected.

 {
     "name": "pipeline4",
     "properties": {
         "activities": [
             {
                 "name": "Azure Data Explorer Command1",
                 "type": "AzureDataExplorerCommand",
                 "dependsOn": [],
                 "policy": {
                     "timeout": "7.00:00:00",
                     "retry": 0,
                     "retryIntervalInSeconds": 30,
                     "secureOutput": false,
                     "secureInput": false
                 },
                 "userProperties": [],
                 "typeProperties": {
                     "command": "asasasa",
                     "commandTimeout": "00:20:00"
                 },
                 "linkedServiceName": {
                     "referenceName": "AzureDataExplorer1",
                     "type": "LinkedServiceReference",
                     "parameters": {
                         "ADXCluster": "asasasas"
                     }
                 }
             }
         ],
         "annotations": [],
         "lastPublishTime": "2021-08-31T06:15:47Z"
     },
     "type": "Microsoft.DataFactory/factories/pipelines"
 }

I have provided a feedback to product team to include UI support for parameters in Azure Data Explorer Command activity and a work item has logged to track it. Hopefully this feature will be available in near feature.


Hope this helps. Do let us know if you have further query.



  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.




· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ericswanson-8494,

We still have not heard back from you. Just wanted to check if the above suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 Votes 0 ·
KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered

Hi @ericswanson-8494,

Just wanted to update you that Product team has implemented a feature to parameterize Kusto linked service for multiple clusters.

147039-image.png

Thank you for sharing your valuable feedback and helping us improve the Azure products.



image.png (187.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.