question

AshutoshYadav-8355 avatar image
0 Votes"
AshutoshYadav-8355 asked KranthiPakala-MSFT commented

ADF pipeline failing for unicode character �

Hi All,

I have pipeline in which I am extracting data from vertica and loading adls in parquet format.

It ran fine till june month data of current year but from july onwards its failing because of the character �
which is present in few columns.

Below are the sample data, can someone help me how to handle this error:

Brice�o Ruiz, Magdalena Constanza
Yeso La Lime�a S.A.C.
Villegas Casta�eda, Diego Nicolas

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @AshutoshYadav-8355,

Thanks for using Microsoft Q&A forum and posting your query.

You have mentioned that pipelines ran fine until June of this year and started failing after that. Could you please confirm if your data before June also contains special character � ? Or only the data after June is having this special character � and pipelines started failing? The reason I would like to confirm this is because if your data before June doesn't have special character then it is for sure data issue (special character) and may have to be handled on source side. But in case if your data before June also contains this special characters and the pipelines ran successful, then it could be something else. It could be related to ODBC driver update too.

Here is a thread where a similar issue was discussed and it is also related to driver update - https://community.powerbi.com/t5/Issues/ODBC-ERROR-HY000-Vertica-Support-50310-Unrecognized-ICU/idi-p/736385.

If this is related to driver update then I would recommend you to file a support ticket. Incase if you don't have a support plan, please do let me know here so that we can work with your offline.

We look forward to your response.

Thank you


  • Please don't forget to click on 130616-image.png and upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.


Hi @AshutoshYadav-8355,

Just checking in to see if the above response was helpful. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

untill june dont see any � though this is data issue pipeline should not fail it should copy the data as it is .
I have tried other etl tool its not failing there but in adf its failing .
� is most likely replacement character for the uknown unicode character.

0 Votes 0 ·

Hi @ashutoshyadav-8355,

Sorry for getting back late on this. I wasn't able to reproduce the problem as I don't have a Vertica instance.

From the error message I can notice that the error might be thrown by the driver. For deeper investigation and analysis, I would recommend you to please file a support ticket so that a support engineer can have a deeper look over a call.

In case if you don't have support please let us know here so that we can work with you offline. In case if you have a support plan and already filed a support ticket, please do share the support request number with us here so that we can track it internally.

We look forward to your confirmation

0 Votes 0 ·

@KranthiPakala-MSFT can you help me where I have to raise ticket and the process or link.

0 Votes 0 ·

@KranthiPakala-MSFT can you help me where I have to raise ticket and the process any link I can follow.

0 Votes 0 ·

Thanks for getting back @AshutoshYadav-8355 .
If you have support plan, you can follow this document to file a support ticket - Create an Azure support request. Please do share the support request number once it if filed.
In case if you don't have a support plan please let us know.

Thank you


0 Votes 0 ·