Partitioning data in ADF while copying Parquet source to Delta sink, based on specific columns

Question

I am trying to partition data on two key columns while transforming parquet source in to a delta sink.

Source (.parquet) - In my file I have Year column that needs to be created as partition column in target side.

/raw/myfolder/myfile.parquet

Target (Delta)

/raw/myfolder/year=<value>/delta_log

-----------------------------
/part*.parquet

I have tried a simple dataflow. But it is NOT partitioning the data.
Even in the _delta_log json file it shows partition info is blank.

• Is transformation to delta file format in preview mode?
• Is this a known issue?
• If this is a known issue, is there a work around?

Answer 1

Hello @shadman44717 ,

Welcome to the Microsoft Q&A platform.

I did encounter a similar behavior. However, I was able to overcome the same by setting the partition at the source end.

The partitioned data was flowing as a stream. I was able to achieve the partitioned data as a result.

Update :

There was a UI issue that was recently fixed by the engineering team. Until the fix reflects, You can change the type of sink to something else, like a delimited text sink, and you should then see the key columns in Key partitioning. Then, switch the Sink type back to Delta.

Reference : https://learn.microsoft.com/en-us/answers/questions/599075/index.html

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators