Azure data factory - errors on columns that has name starting with @ (special characters)

Question

Azure data factory - errors on columns that has name starting with @ (special characters)

We have a csv file, it has first column with column name "@dataetag" .

Azure data factory - fails and gives error on that column. any solution to force process this column?

Answer 1

Hi Grover, Jyoti ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

Could you please share more details as what exactly you have tried in your solution. Are you using copy activity to copy the data from csv source file to the target datastore ? Are you using dataflow ? Where are you exactly getting the error?

I tried to create a sample csv file having first column as '@ID' and copied the file from one container to another using copy activity. It worked without any error.

In case you are trying to copy from csv(ADLS) to SQL table, you could try mapping the column to remove @ in the target like below:

However , if you are getting this error in dataflow, then kindly try removing auto mapping and use select transformation having replace function to replace @ with blank '' in the column name.

Related resources: https://stackoverflow.com/questions/68216473/data-factory-special-character-in-column-headers

https://stackoverflow.com/questions/70107912/how-to-escape-an-in-source-column-in-azure-data-factory

Hope it helps. Looking forward to your response. Thankyou

Answer 2

Hello @Grover, Jyoti

The issue you are facing is because the column name starts with a special character @.

Azure Data Factory does not support column names that start with special characters.

To resolve this issue, you can rename the column in your CSV file to remove the @ symbol. Once you have renamed the column, you can then upload the updated CSV file to your data store and use it in your data factory pipeline.

If you are unable to rename the column in your CSV file, you can try using a data transformation tool like Azure Databricks or Azure HDInsight to preprocess the data and rename the column before ingesting it into Azure Data Factory. This should solve your problem.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.