Data Factory REST Dataset - support 'utf8' charset variant

Bogdan Cupcea 1 Reputation point
2021-05-11T08:50:30.007+00:00

I'm trying to use a REST dataset inside a Copy Data activity in order to call an API and save the response in Data Lake. Unfortunately the API that i'm calling, has the response Content-Type header as "application/json; charset=utf8". As you can see, the charset is utf8 instead of utf-8. I get an error because of that:

Operation on target <activity_name> failed: Failure happened on the 'Source' side. ErrorCode=UserErrorCharSetInvalidOrNotSupported, 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The CharSet 'utf8' is invalid or not supported from Response.Headers.Content-Type,Source=Microsoft.DataTransfer.ClientLibrary." Type=System.ArgumentException,Message='utf8' is not a supported encoding name

Is there any workaround this issue?
Thanks.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,621 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2021-05-13T21:26:13.127+00:00

    Hi @Bogdan Cupcea ,

    As per the discussion with internal team ADF does not include 'utf8 in supported encodings. The reason is utf8 without a hyphen is not supported in .NET. Hence, the official name of this encoding method is strictly "utf-8". Check below for the .NET API that we use for getting the encoding:

    96532-image.png

    So at this time there is no workaround from ADF side but you may have to consider a solution outside of ADF (If possible from your source API).

    Hope this info helps.

    ----------

    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.