dataflow - select distinctrows

Question

field1 field2 field3
name1 surname1 address1
name1 surename1 address1
name1 surename1 address1
name2 surename2 address2
name2 surename2 address2
name2 surename2 address2
...

In my select activity, it returns in data preview several fields.
There are duplicates and I would like to return distict rows.
After the select activity I have placed an aggregate activity.
Inside this aggregate activity the screenshot below

How is this done please?

Answer 1

Hi @arkiboys ,
Thankyou for using Microsoft Q&A platform and posting your query.
As I understand your query, you want to remove duplicate in your data. Your approach of trying aggregate with column pattern seems correct but before that try adding Rank Transformation with Dense option enabled to generate Id based on the field values.

Based on this id , we can group by in group by settings of aggregate transformation. In aggregate settings, we can use column pattern with condition as : name !='Id' . Use Column name expression as : $$ and value expression as : first($$)

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators