Manual template import to data factory

Question

Manual template import to data factory

Anonymous

Hi All,

We have two data factory pipelines using common dataset.

One of the pipeline along with dependent dataset was migrated to different ADF environment.

Now when trying to migrate the second pipeline to the same target environment it is creating new dataset with a suffix “1” at the end of dataset name.

We dont want to have multiple copies of same dataset in target ADF environment.

Cannot delete the existing dataset and rename the new one by removing suffix as this will break the dataset link to the first pipeline.

What is the workaround to this? We expect both the pipelines to reference same dataset in target environment same as the source ADf env.

Note: we are manually exporting and importing pipeline template.

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-16T21:44:18.417+00:00

Hello @Anonymous ,

Welcome to the MS Q&A platform.

Could you please provide me with more details about your migration method of exporting and importing the pipelines?

My understanding is that while importing pipeline templates, you need to create the linked services first (or) select the linked services from the existing list, (or) create a new linked service during the pipeline import process.

Export pipeline:

Import to a different DF:
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-20T17:38:43.483+00:00

Hello @Anonymous ,
I am just checking in to see if you have any updates here.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-22T22:21:47.467+00:00

Hello @Anonymous ,
I am just checking in to see if you have any updates here.
Anonymous

2022-09-22T22:26:43.817+00:00

Hello,

Let me try simplifying this,

lets say we have two pipelines developed in ADF Dev env - PL1 and PL2

Both of these pipelines have a copy activity that uses a common source dataset say DS1

I opened the first pipeline PL1 and did a "export template"

connected to ADF QA env and did a "import from pipeline template"

PL1 and all related objects got deployed to ADF QA without any issues. All good so far

Next when I repeat the same steps to export and import second pipeline PL2, since DS1 is now already present in ADF QA, ADF is creating a new version of it say DS1_1 and mapping this new dataset in the copy activity of PL2.

I expect both the pipelines to continue using same DS1 dataset in ADF QA as well.

If I try to delete DS1_1 dataset after import, it breaks the mapping to PL2 and I have to manually select DS1 and redo the parameter mapping in dataset properties

Hope this helps explain my situation.

Thanks

1 answer

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-16T21:44:18.417+00:00

Hello @Anonymous ,

Welcome to the MS Q&A platform.

Could you please provide me with more details about your migration method of exporting and importing the pipelines?

My understanding is that while importing pipeline templates, you need to create the linked services first (or) select the linked services from the existing list, (or) create a new linked service during the pipeline import process.

Export pipeline:

Import to a different DF:
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-20T17:38:43.483+00:00

Hello @Anonymous ,
I am just checking in to see if you have any updates here.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-22T22:21:47.467+00:00

Hello @Anonymous ,
I am just checking in to see if you have any updates here.
Anonymous

2022-09-22T22:26:43.817+00:00

Hello,

Let me try simplifying this,

lets say we have two pipelines developed in ADF Dev env - PL1 and PL2

Both of these pipelines have a copy activity that uses a common source dataset say DS1

I opened the first pipeline PL1 and did a "export template"

connected to ADF QA env and did a "import from pipeline template"

PL1 and all related objects got deployed to ADF QA without any issues. All good so far

Next when I repeat the same steps to export and import second pipeline PL2, since DS1 is now already present in ADF QA, ADF is creating a new version of it say DS1_1 and mapping this new dataset in the copy activity of PL2.

I expect both the pipelines to continue using same DS1 dataset in ADF QA as well.

If I try to delete DS1_1 dataset after import, it breaks the mapping to PL2 and I have to manually select DS1 and redo the parameter mapping in dataset properties

Hope this helps explain my situation.

Thanks

Answer 1

Bhargava-MSFT 31,261 Microsoft Employee Moderator

Hello @Anonymous ,

Thank you for the detailed explanation.

When you use "Import from pipeline template" for the 1st pipeline, the datasets will publish to the QA environment with the original name.

When you use "Import from pipeline template" for the 2nd pipeline, since the datasets have already been published, ADF creates new datasets with the name dataset1(so you will end up with two datasets)

This is expected behavior.

Workaround:

Option 1)

Export ARM template from Dev and Import the template using custom deployment.

In this case, all resources- pipelines, datasets, linked services, triggers, etc. will be exported into the ARM template.

If you don't want to export all resources, you can edit the template and publish the changes accordingly. I believe this is more aligned with your requirement.

Option 2)

Using source control.

With Git Integration, you can publish all the changes from one environment to another.

In this case, you can't export a single pipeline. All published changes will move to the other environment.

I hope this helps.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-26T18:56:03.673+00:00

Hello @Anonymous ,
I am just checking to see if you have any further questions here.
Anonymous

2022-09-28T03:22:54.857+00:00

Thanks for suggesting the workarounds but not really what I was looking for. Would be much better if ADF just prompts to check during import if user wants to overwrite, ignore or create new version of already existing resources in new environment
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-09-28T17:43:32.817+00:00

Hello @Anonymous ,
Thanks for your feedback. I agree with you. It would be an excellent feature for ADF/Synapse. But unfortunately, this feature is not supported yet in ADF/Synapse.

I recommend submitting a feature request to ADF prompts to check if the user wants to overwrite, ignore or create a new dataset during the import process.

https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

All the feedback shared in this forum is actively monitored and reviewed by the product team, prioritizing them accordingly based on the number of requests.

Once you raise the feature request, please let me know the URL so that I can also up-vote and comment from my end on it. And I will take it forward to the respective product team and provide you with any feedback from them.

Share via

Manual template import to data factory

1 answer

Your answer