delete row in .parquet sink - Delta

arkiboys 9,686 Reputation points
2021-10-04T20:28:46.443+00:00

Hello,
I have the upsert if working for the .parquet files by selecting the upsert if setting to true()For example when the source is update ,then the parquet sink is updated accordingly.But if a row is deleted in source it is still shown in sink parquet even when check the Allow Delete.Any suggestions?Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,075 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2021-10-05T23:33:44.58+00:00

    Hello @arkiboys ,
    Can you please share how you have implement UPSERT in paraquet , Alter row is not supported for paraquet files .
    https://learn.microsoft.com/en-us/azure/data-factory/data-flow-alter-row#allow-alter-row-policies-in-sink

    Allow alter row policies in sink
    For the alter row policies to work, the data stream must write to a database or Cosmos sink. In the Settings tab in your sink, enable which alter row policies are allowed for that sink.

    This is what you see when you select paraquet file

    137922-image.png

    This is what you see when you select a database ( where Alter row is supported )
    137933-image.png

    Please do let me know how it goes .
    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators