CSV to Parquet file conversion

kshitij jain 1 Reputation point
2021-07-01T10:01:26.467+00:00

"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=ParquetInvalidColumnName,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The column name is invalid. Column name cannot contain these character:[,;{}()\n\t=],Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "Master_Table_Parquet",
"details": []

Getting this error while converting csv to parquet using copy activity, I also tried the conversion using data flow and removed the spaces and special characters using select activity. But same error, does that mean that parquet files doesn't support white spaces and special characters in column names ?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,872 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 35,772 Reputation points Microsoft Employee
    2021-07-01T18:48:53.977+00:00

    Hi @kshitij jain ,

    Welcome to Microsoft Q&A forum and thanks for posting your query.

    Yes, you are correct. White space in column name is not supported for Parquet files. Please refer to note in this doc: Parquet format in Azure Data Factory

    Here is an blog posted by an MVP (Disclaimer: Not an MSFT official blog) which is relevant to the issue you are facing and possible solution to overcome it.

    Here is the blog: Export Parquet Files with Column Names with Spaces

    Hope this info helps. Do let us know how it goes.

    ----------

    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.