Filter out Data if already exists in database table using multiple columns

Question

Hi Experts,

I am trying to filter out data from source to destination if it already exists using a combination of columns (they do not form a unique key) and its not working.

example: if data exists for

date: 7/1/2023 for advertiserid 123 with campaign id 456, placementid 789 and keyword id 1011 then do not load this data

Hoping for a better solution.

Answer 1

@AJ Welcome to Microsoft Q&A forum and thanks for reaching out here.

To achieve your requirement, you can use checksums and hash a row fingerprint to filter out the source data based on specific columns.

Related docs: Mapping data flow functions

Here is detailed demonstration of similar use case scenario from ADF product team: ADF: Detect source data changes in Data Flows

Hope this info helps. In case if you have already tried it and having any issues or if your requirement is different than what has been discussed in the demonstration, kindly share few additional details (like sample dummy data for testing) so that we can assist accordingly.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.