Synapse - Rest API Returning Charset "UTF-8" (with the quotes)

Question

Hello,

I'm attempting to do a copy data activity from a Rest API data source, but it seems to be returning a header that says that the charset is "UTF-8" with the quotes, and Synapse is rejecting it.

"message": "Failure happened on 'Source' side. ErrorCode=UserErrorCharSetInvalidOrNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The CharSet '\"utf-8\"' is invalid or not supported from Response.Headers.Content-Type.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentException,Message='\"utf-8\"' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method.\r\nParameter name: name,Source=mscorlib,'"

I don't have control over the API that is being called, so I can't change what is being returned. I'm looking for a way to tell synapse to ignore the charset header and just assume that it's utf-8 regardless. Is that something that exists?

If not, is there something other than the copy activity in synapse that can ignore the headers from an API call and let me edit that header before it tries to process the data?

Thanks.

Answer 1

Hello @SDHorita ,

Thanks for the question and using MS Q&A platform.

Unfortunately, there is no workaround as it is failing at the connector level which means, it is failing even before entering the ADF pipeline. Root cause is as you mentioned the double quote in encoding method which is not supported in .NET.

Here is another thread where a similar issue has been discussed: Data Factory REST Dataset - support 'utf8' charset variant

Ideally this should be handled on API end.

Let us know if you have further questions.

Thank you