ADF pipeline lookup distinct rows from csv

Question

1- lookup1 reads csv rows --> .csv may have duplicate rows
2- foreach1 is fed from lookup1
3- inside foreach1, there is dataflow1 which and copy activity which processes data
...

Question:
How is it possible to get only the distinct rows from .csv and then proceed?
is it possible for the lookup to get distinct rows only?

Thanks

Answer 1

Hey,
Based on my understanding, you would need to use dataflow for that distinct rows since your lookup is a file rather than SQL .
https://www.cloudfronts.com/azure/adfs-mapping-data-flows-how-do-you-get-distinct-rows-and-rows-count-from-the-data-source/

@Nasreen Akter : any thoughts on this ?

Answer 2

Hi @arkiboys , please go through below contents, it might help. Thanks

https://www.youtube.com/watch?v=ryYo8UFUgTI

https://www.youtube.com/watch?v=YDRn_arRtjA

https://eax360.com/find-duplicate-records-with-azure-data-factory-same-source-file/

https://stackoverflow.com/questions/57380501/azure-data-flow-filter-distinct-rows