How to ignore the records in ADF Data Flows

venkat rao 65

Hi All

I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table.
I want to lockup historical data with incremental data transamination and ignore the records coming in the incremental which are less than or equal to TimeStampUpdated
Require your assistance

PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee

2024-06-10T08:32:00.3+00:00

@venkat rao Just checking in to see if the below answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

2 answers

Jing Zhou 7,185 Reputation points Microsoft Vendor

2024-05-23T08:25:36.4666667+00:00

Hello,

Thank you for posting in Q&A forum.

Based on your description, you are building a data conversion process that includes a timestamp field named TimeStampUpdated. You want to lock historical data through incremental data conversion and ignore records in the increment that are less than or equal to TimeStampUpdated. This requirement can usually be achieved through conditional statements in data flow processing. You can set conditions in the data flow to filter out data that meets the conditions, thereby achieving the functions you need.

Best regards，

Jill Zhou

If the Answer is helpful, please click "Accept Answer" and upvote it.
Please sign in to rate this answer.
venkat rao 65 Reputation points

2024-05-23T16:50:55.5766667+00:00

Hi @jing zhou

thanks for your reply , you men to use conditional split transformations
cloud you please provide any examples
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
ShaikMaheer-MSFT 38,456 Reputation points Microsoft Employee

2024-05-23T16:36:44.7066667+00:00

Hi venkat rao,

Thank you for posting query in Microsoft Q&A Platform.

If I understand correctly, using Mapping dataflow you want to perform incremental load by comparing values between source and sink based on "TimeStampUpdated" column. Please correctly If my understanding wrong with more details.

Please fellow below steps to achieve it.

Step1: Source transformation to take source data.

Step2: Another source transformation to take sink data for value in column TimeStampUpdated column.

Step3: For step2 source add sink transformation and use cached sink. Below video explains cached sink.

Cache Sink and Cached lookup in Mapping Data Flow in Azure Data Factory

Step4: For step1 source transformation, add filter transformation and write condition filter rows based TimeStampUpdated column value from cached sink.

Step5: Add sink transformation and load filtered data to sink.

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well.
Please sign in to rate this answer.
venkat rao 65 Reputation points

2024-05-23T17:19:49.1866667+00:00

Hi @ShaikMaheer-MSFT

Thanks for your responses , I have used used dataflows to perform first batch incremental load when it comes to the next batch processing I want to lockup the existing data that was processed in the first batch and ignore records in the next batch increment which are less than or equal to the timestamp column values TimeStampUpdatedIt's
like

I know that we can add another source to lock up the data , I am confused how to ignore records using assert or conditionals transformation in dataflow
Please let me know if you any solutions

ShaikMaheer-MSFT 38,456 Reputation points Microsoft Employee

2024-05-24T05:02:28.4233333+00:00

Hi venkat rao,
yes, you need to use another source transformation and lock up the data.
Assert or any other transformations may not be helpful here. Kindly consider using another source transformation only.
Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

venkat rao 65 Reputation points

2024-05-24T10:15:18.04+00:00

Hi @ShaikMaheer-MSFT ,
Thank you , but my concern was about to archive source1 TimeStampUpdated >= source2 TimeStampUpdated
Kindly let me know how can we archive this ,I know how it has to be
I didn't have any solution yet

ShaikMaheer-MSFT 38,456 Reputation points Microsoft Employee

2024-05-27T05:44:07.1266667+00:00

Hi venkat rao,

you mean to say take data with condition source1 TimeStampUpdated >= source2 TimeStampUpdated and archive it to some back up location?
If yes, you can easily do that using joins. Use above condition in join and then using select transformation to take only source1 columns and then load it to backup location.
Hope this helps. Please let me know if any further.

Please consider hitting Accept Answer button. Accepted answer helps community as well.

ShaikMaheer-MSFT 38,456 Reputation points Microsoft Employee

2024-05-28T08:18:49.66+00:00

Hi venkat rao, Just checking if above answer helps. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to ignore the records in ADF Data Flows

2 answers

Your answer