Is there a way to get resource definition json files of Azure Data Factory via command line?

Question

Is there a way to get resource definition json files of Azure Data Factory via command line?

Makoto Oda 266

I read the Stack Overflow's question which seems to relate to my question.

In the above question, REST API was introduced.

However, I think that there are some differences between REST API and downloading support files from Azure Data Factory Studio.

When I use Linked Services - Get, "id" and "etag" properties are contained in the output.
However, the two properties are not contained in downloaded support files from Azure Data Factory Studio.

If I change the output using jq like | jq 'del(.etag) | del(.id)', I can remove the two properties.
However, the order of properties are not the same as a downloaded support file even if I use jq's --sort-keys option.

For example, when I get json of Azure Key Vault linked service from Azure Data Factory Studio, the content are as follows.

   json  
   {  
       "name": "<LinkedService Name>",  
       "type": "Microsoft.DataFactory/factories/linkedservices",  
       "properties": {  
           "type": "AzureKeyVault",  
           "typeProperties": {  
               "baseUrl": "https://<key container name>.vault.azure.net/"  
           },  
           "annotations": []  
       }  
   }

When I execute REST API, the content are as follows.

command text
curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer <AccessToken>" https://management.azure.com/subscriptions/\<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.DataFactory/factories/<DataFactoryName>/linkedservices/<LinkedServiceName>?api-version=2018-06-01 | jq --sort-keys 'del(.etag) | del(.id)'
output json
{
"name": "<LinkedService Name>",
"properties": {
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://<key container name>.vault.azure.net/"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}

Is there a way to get resouce definition json from command line as same as downloading support files from Azure Data Factory Studio?

MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2022-02-08T20:25:47.757+00:00

Hello @Makoto Oda and welcome to Microsoft Q&A. Thank you for bringing this ask to light. I have retrieved definitions the same way you did, but never before noticed the difference in the results. I see both have the same properties/data but a different structure/organization.
I will try a couple other ways of retrieving for comparison.
Makoto Oda 266 Reputation points

2022-02-10T06:44:03.517+00:00
@MartinJaffer-MSFT Thank you for your reply.

I found additional differences in the pipeline.

When I use REST API ( Pipelines - Get ), the following properties were also returned.

inputs.parameters

outputs.parameters

policy

However, these properties are not in the json data copied from Azure Data Factory Studio.

I upload the following images. (Left: Data Factory Studio sorted by jq --sort-keys, Right: REST API via jq --sort-keys "del(.etag) | del(.id)"

I hope that two outputs are the same.
MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2022-02-14T19:14:49.973+00:00

I noticed the "lastPublishTime" in your analysis. I didn't particularly remember seeing this before so I took a deeper look. Publish Time isn't something typically of use when authoring. I pulled up one of my Git-enabled factories and compared the same pipeline in Factory mode and Git mode. Only Factory mode had "lastPublishTime". This is expected.

You do not need to worry about the outputs.parameters:[] . I believe this is created in the publish process to preclude the possibility of a "property not found" error by any consuming service. Indeed it is not found in the left pane as shown, and by default its value would be null or empty.

I then did the same with a KeyVault linked service. After sorting, the only difference was the "type": "Microsoft.DataFactory/factories/linkedservices", present in the Factory mode's definition.

I'm not sure if your Factory has a repository integrated or not, but I can tell you a few things. All SDK, REST API, Cli, fetch directly from the Factory mode. They do not interact with any repository you have integrated.
Makoto Oda 266 Reputation points

2022-02-22T00:26:11.38+00:00

Thank you for your reply. I do not integrate a repository.

I have not seen "Factory mode" and "Git mode" in Azure Documents.
Please tell me an article which these concepts are written in.

And, I cannot understand why the outputs.parameters:[] does not exist in the Azure Data Factory Studio downloaded support files.

To preclude the possibility of a "property not found" error as you wrote, I think that the support files also should contain the outputs.parameters:[].
Makoto Oda 266 Reputation points

2022-04-01T06:52:33.017+00:00

@MartinJaffer-MSFT

Is there any progress...?

MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2022-02-08T20:25:47.757+00:00

Hello @Makoto Oda and welcome to Microsoft Q&A. Thank you for bringing this ask to light. I have retrieved definitions the same way you did, but never before noticed the difference in the results. I see both have the same properties/data but a different structure/organization.
I will try a couple other ways of retrieving for comparison.
Makoto Oda 266 Reputation points

2022-02-10T06:44:03.517+00:00

@MartinJaffer-MSFT Thank you for your reply.

I found additional differences in the pipeline.

When I use REST API ( Pipelines - Get ), the following properties were also returned.

inputs.parameters

outputs.parameters

policy

However, these properties are not in the json data copied from Azure Data Factory Studio.

I upload the following images. (Left: Data Factory Studio sorted by jq --sort-keys, Right: REST API via jq --sort-keys "del(.etag) | del(.id)"

I hope that two outputs are the same.
MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2022-02-14T19:14:49.973+00:00

I noticed the "lastPublishTime" in your analysis. I didn't particularly remember seeing this before so I took a deeper look. Publish Time isn't something typically of use when authoring. I pulled up one of my Git-enabled factories and compared the same pipeline in Factory mode and Git mode. Only Factory mode had "lastPublishTime". This is expected.

You do not need to worry about the outputs.parameters:[] . I believe this is created in the publish process to preclude the possibility of a "property not found" error by any consuming service. Indeed it is not found in the left pane as shown, and by default its value would be null or empty.

I then did the same with a KeyVault linked service. After sorting, the only difference was the "type": "Microsoft.DataFactory/factories/linkedservices", present in the Factory mode's definition.

I'm not sure if your Factory has a repository integrated or not, but I can tell you a few things. All SDK, REST API, Cli, fetch directly from the Factory mode. They do not interact with any repository you have integrated.
Makoto Oda 266 Reputation points

2022-02-22T00:26:11.38+00:00

Thank you for your reply. I do not integrate a repository.

I have not seen "Factory mode" and "Git mode" in Azure Documents.
Please tell me an article which these concepts are written in.

And, I cannot understand why the outputs.parameters:[] does not exist in the Azure Data Factory Studio downloaded support files.

To preclude the possibility of a "property not found" error as you wrote, I think that the support files also should contain the outputs.parameters:[].
Makoto Oda 266 Reputation points

2022-04-01T06:52:33.017+00:00

@MartinJaffer-MSFT

Is there any progress...?

Is there a way to get resource definition json files of Azure Data Factory via command line?

Activity