ADF pipeline failing for unicode character �

Ashutosh Yadav 1 Reputation point
2021-10-12T11:48:37.163+00:00

Hi All,

I have pipeline in which I am extracting data from vertica and loading adls in parquet format.

It ran fine till june month data of current year but from july onwards its failing because of the character �
which is present in few columns.

Below are the sample data, can someone help me how to handle this error:

Brice�o Ruiz, Magdalena Constanza
Yeso La Lime�a S.A.C.
Villegas Casta�eda, Diego Nicolas

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,481 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2021-10-12T21:22:06.14+00:00

    Hi @Ashutosh Yadav ,

    Thanks for using Microsoft Q&A forum and posting your query.

    You have mentioned that pipelines ran fine until June of this year and started failing after that. Could you please confirm if your data before June also contains special character � ? Or only the data after June is having this special character � and pipelines started failing? The reason I would like to confirm this is because if your data before June doesn't have special character then it is for sure data issue (special character) and may have to be handled on source side. But in case if your data before June also contains this special characters and the pipelines ran successful, then it could be something else. It could be related to ODBC driver update too.

    Here is a thread where a similar issue was discussed and it is also related to driver update - https://community.powerbi.com/t5/Issues/ODBC-ERROR-HY000-Vertica-Support-50310-Unrecognized-ICU/idi-p/736385.

    If this is related to driver update then I would recommend you to file a support ticket. Incase if you don't have a support plan, please do let me know here so that we can work with your offline.

    We look forward to your response.

    Thank you

    ----------

    • Please don't forget to click on 130616-image.png and upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators