Converting csv to parquet is not populating correct values in the columns in parquet file

Question

Hi Team,
I am trying to convert a simple .csv files which has lot of columns (~161 columns) into parquet file using mapping dataflow. All the columns show correct values when doing a 'Data Preview' but when the dataflow is run, the generated parquet file has drifted values in some of the columns which is wrong. Not sure what is happening on the destination side.
For destination setting, we are using 'Use Current Partition'
Could you help us understand where we are going wrong?

Answer 1

Hi Madugundu Somashekara, Roopa:

Thanks for reaching out to Microsoft Q&A.

I see a similar question from you has been answered by mark and kranthi. While that might not be the solution for this issue, it proves that csv doesn't maintain data types.

Did you try the same options and still you received these value mismatch error?

https://learn.microsoft.com/en-us/answers/questions/856132/converting-csv-to-parquet-format-with-correct-data

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Answer 2

Hi Madugundu Somashekara, Roopa ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you are trying to convert csv into parquet using dataflow in adf pipeline but not getting expected results .

Could you please try using 'map drifted' option in the data preview tab .

Go to the Data Preview tab and click Refresh to fetch a data preview. If data factory detects that drifted columns exist, you can click Map Drifted and generate a derived column that allows you to reference all drifted columns in schema views downstream.

For more details, kindly check this documentation: Map drifted columns quick action

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou