i want load table data into multiple files based on record count

Question

i want to load table data into multiple files using ADF.

ex:in a table 1000 records so i want load first 1-100 records into 1 file and second 101-200 in the second file and so on. how can i achieve this using ADF.

Answer 1

Hi,

1. You can use a script activity to find out number of rows in your table.
2. Add a data flow, with a integer parameter again called rowcount
3. Pass the rowcount from variable to the data flow parameter as shown (Click on data flow activity, and go to parameters tab and click on rowcount parameter. then Choose pipeline expression and provide the expression given as below)
4. click on Optimize tab in data flow sink and give the settings as shown. this will make sure to split your table rows as per your ask.
5. If you dont want to give default names to the partitioned files, use below setting
6. Just one thing to note, Round robin doesnt put first 100 in one partition, but it splits rows in round robin fashion. Please try these and let us know. thanks

Answer 2

any update