Re-deployment of Data factory resource makes publish process messy

asked 2022-06-05T20:14:20.37+00:00
Janne Kujanpää 106 Reputation points

I have added role assignments with my data factory template and therefore I sometimes need to re-deploy data factory resource.

Repo configuration(1) includes property called lastCommitId that is not part of my template.

LastCommitId is updated by data factory itself and I do not use that property on my template. As soon as I re-deploy my template lastCommitId property is removed:

208535-image.png

then running ADF instance thinks that it must re-create "all" objects when publishing from the collaboration branch:
208529-image.png

I tried well-known workarounds in my template:

  • lastCommitId: json('null')
  • lastCommitId: null

but both workarounds end up deleting the lastCommitId property instead of ignoring the value change.

This makes publish process really messy with larger projects after Data factory resource re-deployment.

Is there any good workarounds for this?

I can think of at least two workarounds:

  • Capture lastCommitId before deployment and then set it back
  • or capture lastCommitId before deployment and pass it as a template parameter

Any better workarounds without adding complexity to the template/pipeline?

EDIT:
As a workaround, I read the value of lastCommitId in pre-commit scripts and inject the value as template parameter value into the deployment script. That is working for me but I would like not to add extra steps to maintain the idempotency of the deployment templates.

(1) https://learn.microsoft.com/en-us/azure/templates/microsoft.datafactory/factories?tabs=bicep#factoryrepoconfiguration

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,260 questions
{count} votes