Filter out Data if already exists in database table using multiple columns

AJ 0 Reputation points
2023-08-01T01:19:49.86+00:00

Hi Experts,

I am trying to filter out data from source to destination if it already exists using a combination of columns (they do not form a unique key) and its not working.

example: if data exists for

date: 7/1/2023 for advertiserid 123 with campaign id 456, placementid 789 and keyword id 1011 then do not load this data

Hoping for a better solution.

User's image

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,133 questions
{count} votes

1 answer

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,407 Reputation points Microsoft Employee
    2023-08-01T23:42:22.1933333+00:00

    @AJ Welcome to Microsoft Q&A forum and thanks for reaching out here.

    To achieve your requirement, you can use checksums and hash a row fingerprint to filter out the source data based on specific columns.

    Related docs: Mapping data flow functions

    User's image

    Here is detailed demonstration of similar use case scenario from ADF product team: ADF: Detect source data changes in Data Flows

    Hope this info helps. In case if you have already tried it and having any issues or if your requirement is different than what has been discussed in the demonstration, kindly share few additional details (like sample dummy data for testing) so that we can assist accordingly.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments