Share via

Azure datafactory dataflow issue while reading non-english characters from Azure SQL DB

Sudipta Kumar Dutta 6 Reputation points
2022-03-29T22:01:03.32+00:00

I am using a mapping dataflow in data factory to read data from a table in Azure SQL DB database. There are some non-english characters( Japanese, Chinese etc ) in few of the rows. All the rows are read correctly except rows where the non-english characters are there. Some junk characters are coming while reading those non-english rows from the table's column and loading into the target table.

We are loading the stage table ( in Azure SQL DB ) from Oracle using copy activity. Those rows having non-english characters are loaded fine using copy job.

Then when I am using a mapping dataflow from this stage table ( of Azure SQL DB ) to load another table then I am getting this issue. It seems that mapping dataflow is not able recognize the non-english characters correctly.

Please advise.

Azure SQL Database
Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Alberto Morillo 35,506 Reputation points MVP Volunteer Moderator
    2022-03-29T23:17:07.933+00:00

    Please make sure you are using UTF-8 or UTF-16 collations on Azure SQL database to support languages Chinese, Japanese, or Korean characters. Please read here for more details.

    Use UTF-8 Encoding on the Connection settings of the Copy Activity.

    Was this answer helpful?

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.