question

MadugunduSomashekaraRRoopa-1271 avatar image
0 Votes"
MadugunduSomashekaraRRoopa-1271 asked KranthiPakala-MSFT answered

Converting csv to parquet format with correct datatypes

Hi,

I am facing issue while converting the .csv file to parquet files.

As per my requirement, I've converted the parquet to .csv format and added two new columns of string datatype. Now I'm trying to convert this .csv file back to parquet format with original parquet file datatypes using mapping data flows but the datatype conversion in not happening. All the column datatypes are shown as string only.
Is there any workaround to handle this scenario so that my final parquet file shows up the correct data types?

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Text delimited files (CSV) do not maintain data types, everything becomes a string. To apply data types, use the "Detect data types" button on the source projection tab.

0 Votes 0 ·

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered

Hello @MadugunduSomashekaraRRoopa-1271,

Thanks for the question and using MS Q&A platform.

As @MarkKromer-MSFT called out, in your source transformation, under Projection you will see Detect data type button which can be used to auto detect the datatypes from your delimited text files. In case if you want to manually update it to some other data type other than autodetected, you will have a drop-down menu to choose from. In addition, you can also specify a format which is optional.

For more clarification, please see below GIF:

203836-detectdatatypeforsourcetxtdata.gif


To explore about these features, please refer to this product documentation: ADF Mapping Dataflow - Source Transformation - Projection Tab

Here is a related helpful video demonstration by @MarkKromer-MSFT for shaping large dataset projections - Shape large dataset projections w/ADF Select transformation

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.