ADF Https linked service doesn't work with Data Flow

60185333 20 Reputation points
2023-05-17T16:27:27.5933333+00:00

Hello!

We've been trying to create a pipeline to call an API endpoint. I couldn't use Rest API linked service because it doesn't allow for client certificates (.pfx with a password). The only linked service that worked was a HTTP linked service.

We used that and built our pipeline. Then, when we got the output, we got just 1 level down in the hierarchy ( so, for example, we have people ID's but within each person, there's an array with multiple values assigned to each person like positions). So it works to get all the people, but not to go deeper in the nested json.

All the solutions on the internet suggest using Data Flow for more complex transformations, especially with nested json with so many levels. However, Data Flow doesn't seem to work with HTTP linked service.

can you suggest a workaround for this? Is there a solution, which we can use with Data Flow to get all the levels of nested json, and that will take client certificates with a password in the linked service?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,111 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,437 Reputation points Microsoft Employee
    2023-05-19T23:52:23.39+00:00

    Hi @60185333 ,

    Welcome to Microsoft Q&A forum and thanks for reaching out here.

    Yes, HTTP is not supported as source in Data flows and hence you won't be able to use it directly to flatten the complex JSON data from your HTTP Endpoint.

    But as a workaround what you can do is to use the HTTP linked service to call the API endpoint and retrieve the JSON data to a blob storage as staging location using a copy activity, and then use the Data Flow activity next to copy activity and point data flow source to the HTTP response json file to transform the complex JSON data and extract the nested JSON values as needed.

    Once you have the HTTP response data in storage blob, then in mapping data flow, you will have to use Flatten transformation to flatten the nested values.

    Hope this helps.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful