CSV to Parquet file conversion

Question

CSV to Parquet file conversion

kshitij jain 1

"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=ParquetInvalidColumnName,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The column name is invalid. Column name cannot contain these character:[,;{}()\n\t=],Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "Master_Table_Parquet",
"details": []

Getting this error while converting csv to parquet using copy activity, I also tried the conversion using data flow and removed the spaces and special characters using select activity. But same error, does that mean that parquet files doesn't support white spaces and special characters in column names ?

1 answer

Your answer

Answer 1

KranthiPakala-MSFT 46,627 Microsoft Employee

Hi @kshitij jain ,

Welcome to Microsoft Q&A forum and thanks for posting your query.

Yes, you are correct. White space in column name is not supported for Parquet files. Please refer to note in this doc: Parquet format in Azure Data Factory

Here is an blog posted by an MVP (Disclaimer: Not an MSFT official blog) which is relevant to the issue you are facing and possible solution to overcome it.

Here is the blog: Export Parquet Files with Column Names with Spaces

Hope this info helps. Do let us know how it goes.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2021-07-06T17:09:37.82+00:00

Hi @kshitij jain ,

Just checking in to see if the above suggestion was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2021-07-07T16:54:56.297+00:00

Hi @kshitij jain ,

We still have not heard back from you. Just wanted to check if the above information was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Share via

CSV to Parquet file conversion

1 answer

Your answer