question

shadman44717-4729 avatar image
0 Votes"
shadman44717-4729 asked SathyamoorthyVijayakumar-MSFT edited

Partitioning data in ADF while copying Parquet source to Delta sink, based on specific columns

I am trying to partition data on two key columns while transforming parquet source in to a delta sink.

Source (.parquet) - In my file I have Year column that needs to be created as partition column in target side.

/raw/myfolder/myfile.parquet

Target (Delta)

/raw/myfolder/year=<value>/delta_log
-----------------------------/part*.parquet


139091-capture.jpg

I have tried a simple dataflow. But it is NOT partitioning the data.
Even in the _delta_log json file it shows partition info is blank.

  • Is transformation to delta file format in preview mode?

  • Is this a known issue?

  • If this is a known issue, is there a work around?




azure-data-factory
capture.jpg (44.3 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SathyamoorthyVijayakumar-MSFT avatar image
0 Votes"
SathyamoorthyVijayakumar-MSFT answered SathyamoorthyVijayakumar-MSFT edited

Hello @shadman44717-4729,

Welcome to the Microsoft Q&A platform.

I did encounter a similar behavior. However, I was able to overcome the same by setting the partition at the source end.

140315-image.png

The partitioned data was flowing as a stream. I was able to achieve the partitioned data as a result.

Update :


There was a UI issue that was recently fixed by the engineering team. Until the fix reflects, You can change the type of sink to something else, like a delimited text sink, and you should then see the key columns in Key partitioning. Then, switch the Sink type back to Delta.

Reference : https://docs.microsoft.com/en-us/answers/questions/599075/index.html


Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


    [2]: /answers/storage/attachments/140334-image.png

    [3]: /answers/storage/attachments/140333-image.png

    [4]: /answers/storage/attachments/140335-image.png

    [5]: /answers/storage/attachments/140336-image.png



image.png (32.1 KiB)
image.png (26.5 KiB)
image.png (16.0 KiB)
image.png (11.9 KiB)
image.png (15.9 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.