ADF - Special character problem with encoding UTF-8 WITHOUT BOM in DATASET DelimitedText

Question

ADF - Special character problem with encoding UTF-8 WITHOUT BOM in DATASET DelimitedText

Anonymous

I have a case where I have to copy data from a source SQL to a csv with UTF-8 WITHOUT BOM or Default(UTF-8) encoding, but the problem occurs when I visualize the data in the csv, I realize that it is not generating the special characters.

So I turn to you Community to find out a solution to this problem.

Since I did a test with the UTF-8 encoding and if it generates the special characters in the csv but the problem is that it has BOM.

How can I solve this problem?

Thanks for you help!!

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-25T13:59:32.57+00:00

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

I did a quick testing at my end. If my understanding is correct, you are performing a copy activity from SQL DB to a Deleimited text (csv -UTF - 8 without BOM).

But I was not able to observe the issue at my end. Can you please let me know if I m missing something at my end ?

Source SQL

Sink Dataset Configuration :

I initiated the Copy Activity and observed the Output

152560-image.png

I have attached the converted csv ( i had renamed as txt)
Anonymous

2021-11-25T14:41:55.91+00:00

Hello @svijay-MSFT ,

Your understanding of the situation is correct, and your test is the same as I did, but for a strange reason when I check the csv it changes the characters to those of the image that I have in the question.

If in the encoding I select the one that only says UTF-8 if it respects the special characters, but it generates the csv with BOM.

This is the correct way that it should give me the characters:

Do you know with whom I can report this anomaly?
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-26T15:17:12.36+00:00

Hello @Anonymous ,

That's really strange. In ideal scenario, Text without BOM removes only starting bytes of the file and should not affect the content of the file. just to be double sure - when you open the attached text file are you able to see the accents ? This is to eliminate the issue with the editor that use - or you can open the your output file at the portal level.

And, if the behavior is persistent across the text editors - this would need a closer look . I'd recommend you to open a support ticket.

Note :
Also, I see you have dynamic expression for a column delimiter- if you can explain what it is - i can give it a shot at my end as well.
Alternatively, you could have static column delimiter and see whether that is making any difference

Your answer

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-25T13:59:32.57+00:00

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

I did a quick testing at my end. If my understanding is correct, you are performing a copy activity from SQL DB to a Deleimited text (csv -UTF - 8 without BOM).

But I was not able to observe the issue at my end. Can you please let me know if I m missing something at my end ?

Source SQL

Sink Dataset Configuration :

I initiated the Copy Activity and observed the Output

152560-image.png

I have attached the converted csv ( i had renamed as txt)
Anonymous

2021-11-25T14:41:55.91+00:00

Hello @svijay-MSFT ,

Your understanding of the situation is correct, and your test is the same as I did, but for a strange reason when I check the csv it changes the characters to those of the image that I have in the question.

If in the encoding I select the one that only says UTF-8 if it respects the special characters, but it generates the csv with BOM.

This is the correct way that it should give me the characters:

Do you know with whom I can report this anomaly?
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-11-26T15:17:12.36+00:00

Hello @Anonymous ,

That's really strange. In ideal scenario, Text without BOM removes only starting bytes of the file and should not affect the content of the file. just to be double sure - when you open the attached text file are you able to see the accents ? This is to eliminate the issue with the editor that use - or you can open the your output file at the portal level.

And, if the behavior is persistent across the text editors - this would need a closer look . I'd recommend you to open a support ticket.

Note :
Also, I see you have dynamic expression for a column delimiter- if you can explain what it is - i can give it a shot at my end as well.
Alternatively, you could have static column delimiter and see whether that is making any difference

Share via

ADF - Special character problem with encoding UTF-8 WITHOUT BOM in DATASET DelimitedText

Your answer