Dynamic Assertions in Azure Data Factory Dataflow

Question

Dynamic Assertions in Azure Data Factory Dataflow

Abdullah Humayun 40

How can I create dynamic Assertion Rules in Azure Data Factory Dataflow? I am attempting to read a table from a path and create assertion rules for each column dynamically. However, it seems like I can only define assertion rules beforehand and not change them during runtime. Passing column names to the dataflow is a possible solution but it could be costly and time-consuming.

Is there a way to create dynamic assertion rules that can check for uniqueness for each column in the table, for example, when reading a table with 3 columns?

QuantumCache 20,681 Reputation points Moderator

2023-06-07T17:10:19.83+00:00

Hello @Abdullah Humayun

For each new column, did you try to use a dynamic expression to check for uniqueness?
Abdullah Humayun 40 Reputation points

2023-06-08T07:47:41.92+00:00
Hi, so I tried doing that to check for nulls, seems to work as I can create a boolean column that checks for nulls, uniqueness etc

I tried to take it a step further and make it a bit dynamic by passing a map of columns with rules so I can dynamically generate derived columns (eg ["col1"->"NullCheck","col2"->"NullCheck"]). My derived column expression starts to fail with key not found errors when I try to access map values. Have attached the expression below

iif(contains(keys($rulemap),toString(#item) == name),at($rulemap, name) == 'NullCheck',false())
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-06-19T06:20:06.3266667+00:00

Hi Abdullah Humayun, Could you please share your implementation details little more with screenshots and transformations configurations? To understand better and help here. Thank you.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-06-26T10:46:02.52+00:00

Hi Abdullah Humayun, Just checking if you get chance to share updates. Thank you.