Re-deployment of Data factory resource makes publish process messy

asked 2022-06-05T20:14:20.37+00:00
Janne Kujanpää 106 Reputation points

I have added role assignments with my data factory template and therefore I sometimes need to re-deploy data factory resource.

Repo configuration(1) includes property called lastCommitId that is not part of my template.

LastCommitId is updated by data factory itself and I do not use that property on my template. As soon as I re-deploy my template lastCommitId property is removed:


then running ADF instance thinks that it must re-create "all" objects when publishing from the collaboration branch:

I tried well-known workarounds in my template:

  • lastCommitId: json('null')
  • lastCommitId: null

but both workarounds end up deleting the lastCommitId property instead of ignoring the value change.

This makes publish process really messy with larger projects after Data factory resource re-deployment.

Is there any good workarounds for this?

I can think of at least two workarounds:

  • Capture lastCommitId before deployment and then set it back
  • or capture lastCommitId before deployment and pass it as a template parameter

Any better workarounds without adding complexity to the template/pipeline?

As a workaround, I read the value of lastCommitId in pre-commit scripts and inject the value as template parameter value into the deployment script. That is working for me but I would like not to add extra steps to maintain the idempotency of the deployment templates.


Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,260 questions
{count} votes