Share via

Invoking RESI API and Uploading File through API Management using ADF Data Pipeline

Jayaprakash Kumarasamy 1 Reputation point
2022-03-29T02:29:04.047+00:00

Hello,

We have a requirement to Invoke a REST API call through API Management using ADF Data Pipeline. Data Pipeline would have the source as ADLS Gen2 where CSV files would be extracted and invoke External REST API call through API Management API, API Management would act as Façade for the external API call. While invoking the external REST API call File extracted from ADLS should be sent as parameter. Please let me know how we could achieve the requirement and suggest if there are alternative option to achieve the requirement.

Azure API Management
Azure API Management
An Azure service that provides a hybrid, multi-cloud management platform for APIs.
1,803 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,370 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,748 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. James Longworth 326 Reputation points
    2022-03-30T14:52:51.14+00:00

    Thanks,

    So the external API is expecting the actual file content to be sent to it as part of the call? If this is the case then my initial idea won't work and I'm not sure how to proceed, but I provide my initial idea below in case it helps:

    The call would be made to the APIM facade endpoint including a filepath as a parameter. APIM would then call another internal API (such as a function app) which would then use that filepath parameter (along with other information that it may have from config) to generate a user-delegation SAS (readonly, for that specific BLOB) via a system managed identity. This API would then pass back a full SAS based URL to APIM.. This SAS URL would be sent as a parameter to the external API call (by APIM), and the external API would use that SAS based URL to directly access the storage account to download the file.

    There is example MS code for such a function app. The SAS would be given a short lifetime and could even be limited by source IP address (If that would be known in advance). Calling other APIs in APIM policy is done via the 'Send Request' policy. And receiving the returned value and then constructing the call to the down-stream API is straightforward in APIM policy.

    I have implemented almost exactly this to allow end-user client to securely access and download a BLOB from storage, directly, without having to serialise/deserialise the file in the applications.

    Good luck!