ADF Pipeline Error

Question

Hi,
We have a scenario to load the file from file server to ADLS Gen2. While copy the file and storing it in Parquet format getting the below error. Please suggest are we missing something.

One of the column header in the file has the value "(EURO)" symbol.

Failure happened on 'Sink' side. ErrorCode=ParquetInvalidColumnName,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The column name is invalid. Column name cannot contain these character:[,;{}()\n\t=],Source=Microsoft.DataTransfer.Common,'

Answer 1

Hi,
Apologize for the delayed response. I cannot change the column header as it has special character it was throwing error while saving it in parquet format. Addressed this issue by following the below approach (anyone approach can fit).

Solution 1:
In the Dataflow used the activity to select the columns what is required for next layer, over there in Mapping tab have used the mapping like this Column name= position==1 Name as = 'Required column name'.

Solution 2:
In the copy activity, use import schema option under mapping tab and rename the columns with single quote. This also worked.

Answer 2

Hey,
Can you share the list of all header names?
Because parquet file doesn't support white space and special characters in its headers.
So based on the error message it is clear issue w.r.t headers