CSV to Parquet file conversion

Question

"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=ParquetInvalidColumnName,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The column name is invalid. Column name cannot contain these character:[,;{}()\n\t=],Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "Master_Table_Parquet",
"details": []

Getting this error while converting csv to parquet using copy activity, I also tried the conversion using data flow and removed the spaces and special characters using select activity. But same error, does that mean that parquet files doesn't support white spaces and special characters in column names ?

Answer 1

Hi @kshitij jain ,

Welcome to Microsoft Q&A forum and thanks for posting your query.

Yes, you are correct. White space in column name is not supported for Parquet files. Please refer to note in this doc: Parquet format in Azure Data Factory

Here is an blog posted by an MVP (Disclaimer: Not an MSFT official blog) which is relevant to the issue you are facing and possible solution to overcome it.

Here is the blog: Export Parquet Files with Column Names with Spaces

Hope this info helps. Do let us know how it goes.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.