dataflow - select distinctrows

arkiboys 9,646 Reputation points
2022-03-15T16:07:00.52+00:00

field1 field2 field3
name1 surname1 address1
name1 surename1 address1
name1 surename1 address1
name2 surename2 address2
name2 surename2 address2
name2 surename2 address2
...

In my select activity, it returns in data preview several fields.
There are duplicates and I would like to return distict rows.
After the select activity I have placed an aggregate activity.
Inside this aggregate activity the screenshot below

How is this done please?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,721 questions
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 31,561 Reputation points Microsoft Employee
    2022-03-16T17:48:03.5+00:00

    Hi @arkiboys ,
    Thankyou for using Microsoft Q&A platform and posting your query.
    As I understand your query, you want to remove duplicate in your data. Your approach of trying aggregate with column pattern seems correct but before that try adding Rank Transformation with Dense option enabled to generate Id based on the field values.
    183846-image.png

    Based on this id , we can group by in group by settings of aggregate transformation. In aggregate settings, we can use column pattern with condition as : name !='Id' . Use Column name expression as : $$ and value expression as : first($$)

    183775-image.png

    183798-image.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful