Share via

Filter Transformation

Mo 21 Reputation points
2022-01-14T15:53:13.643+00:00

Hi

In mapping dataflows, I'm trying to filter out from the SensitiveRecord all 'Yes', knowing that the other possibilities are 'No', 'Allow' & Null, the only outcome is the 'No'. knowing that Im sinking to a CSV file (Records +1.2m)

in Filter expression I used: SensitiveRecord != 'Yes'

Could you please tell me where I went wrong!
Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Answer accepted by question author
  1. MarkKromer-MSFT 5,231 Reputation points Microsoft Employee Moderator
    2022-01-14T18:59:19.613+00:00

    Are you filtering out the records or sorting? If you want to remove the rows, use the Filter transformation with the expression SensitiveRecord != 'Yes'.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator
    2022-01-17T06:56:13.327+00:00

    Hi @Mo ,

    Thank you for posting query in Microsoft Q&A Platform.

    If your Filter transformation is not able to give you back other values as you mentioned in followup ask. Then consider below work around.

    Step1: My Source has 'Yes', 'No', null(empty value from csv) & 'Allow' values in SensitiveRecord Column.
    165450-image.png

    Step2: Use derived column transformation to identify null rows and update them with some dummy data. In my case I am updating them as 'nullvalue'.
    165631-image.png

    Step3: Use Filter transformation with your expression.
    165543-image.png

    Please note, When you have huge number of rows there is likely a possibility that you may not able to see all possible rows in your data preview.

    Hope this will help. Please let us know if any further queries.

    -----------------

    Please consider hitting Accept Answer button. Accepted answers helps community as well.

    2 people found this answer helpful.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.