Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
Hi @Obaid UrRehman , in the DataFlow, please do 'SORT' before 'SINK'. Thanks!
Data flow sink does not copies data as shown by data preview
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 65%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOU%3C/text%3E%3C/svg%3E)
Obaid UrRehman
166
Reputation points
Hi,
I have a data flow sink that writes data to an inline dataset and creates file on blob.
The dataflow goes through many transformations and I have the desired output in the end before sink.
Here is how the output looks in sink data preview:
but when I download the data from azure blob to local system and open it in excel, it is all messed up and looks like this:
It seems that the data in not in order as shown in the data preview
Here is the setting of the sink:
Appreciate your help! Thanks :)
Accepted answer
-
Nasreen Akter 10,791 Reputation points2022-03-25T00:48:38.74+00:00 -
Obaid UrRehman 166 Reputation points
2022-03-25T09:48:14.863+00:00 I have tried sorting, its the same, every time I run the pipeline the results are in a different order. Do you have any clue why the results are written are different than what are shown in data preview? And what is causing this? Could this be related to partitioning, it should not be because I am using a single partition
Also, When I click the Export CSV button in data preview:
The data in the downloaded csv file is correct but missing an ID column.
-
Obaid UrRehman 166 Reputation points
2022-03-25T14:18:15.323+00:00 Hi @Nasreen Akter ,
I have tried everything, the output doesn't seem to be in the right order. Is it possible for you to help with this please :) I am not sure why is this happening!
Thanks,
Obaid -
Nasreen Akter 10,791 Reputation points
2022-03-25T14:30:06.4+00:00 Hi @Obaid UrRehman , would you please share the
SORTconfigurations andData Previewwith us. Also please check the optionSort only within Partition. Thanks -
Obaid UrRehman 166 Reputation points
2022-03-25T14:37:48.15+00:00 -
Obaid UrRehman 166 Reputation points
2022-03-25T15:17:23.037+00:00 And the sink configuration is:
-
Obaid UrRehman 166 Reputation points
2022-03-25T18:19:51.473+00:00 Hi,
This output was resulting from a join. Refer to this question: https://learn.microsoft.com/en-us/answers/questions/785948/combine-columns-from-two-sources.html
There was some problem with the join, may be because I could explain my problem properly. There is a column month that is common in both datasets and I removed the Surrogate key and added month as sort in Window key in Dataset2. After that the merge was okay and the output created by sink was also fine.
Thanks for the help :)
-
Nasreen Akter 10,791 Reputation points
2022-03-25T18:22:57.653+00:00 @Obaid UrRehman , awesome! Glad to know that you figured out the issue :)
Sign in to comment -
