Extract Data from REST API w/ OAuth2 Authorization Code Grant Type

Question

Hi,

We are trying to extract data from a REST API source that uses OAuth2.0 with grant type 'Authorization Code'. More specifically, we are looking to extract the data of multiple endpoints / tables in parallel and store this in Azure Blob Storage for ingestion in Power BI.

To this end, we had a look at Azure Data Factory but the HTTP & REST connectors do not support this specific grant type. As a result, I only see three options.

Build a custom authorization flow in ADF
In doing this, I am seeing a challenge as the extraction process (for some tables) might take longer than the time the tokens are valid (10 mins). Additionally, we are dealing with rather strict API limits (max. 60 calls / minute).
Therefore I am not sure if this is even a viable path to pursue.

API Gateway

There might be a resource in Azure that is capable of connecting to the REST API with the specified grant type. Perhaps we can then connect to this Azure resource using a natively supported authorization method in ADF. The authorization to the source system is then taken care of in the gateway instead of in Azure.

Other resource

There might be a different resource in Azure than ADF that is perfectly suited for this purpose. Preferably, the resource should be low/no code just like ADF.

We would like to know if there are other people dealing with this challenge and if they have been able to come to a proper solution. Any suggestions and tips are welcome.

Best regards,

Lars

Answer 1

Hi @Lars Peeters | Bryn
Welcome to Microsoft Q&A platform and thanks for posting your question here.

As per my understanding, you are trying to export data from a REST API source that uses OAuth2.0 with grant type 'Authorization Code' and store it in Azure Blob Storage for ingestion in Power BI. You have tried Azure Data Factory, but the HTTP & REST connectors do not support this specific grant type. You also considered building a custom authorization flow in ADF, but you are concerned about the token validity and API limits. Additionally, you are exploring the possibility of using an API Gateway or other Azure resources that are better suited for this purpose.

Based on your requirements, I would suggest using Azure Logic Apps. Azure Logic Apps is a low-code platform that allows you to create workflows that integrate with various services, including REST APIs. Logic Apps has built-in connectors for OAuth2.0 authentication, including the Authorization Code grant type. You can use the HTTP connector to call the REST API endpoints and store the data in Azure Blob Storage using the Azure Blob Storage connector. Logic Apps also supports parallel execution of tasks, which can help you extract data from multiple endpoints/tables in parallel.

To get started with Azure Logic Apps, you can follow the steps outlined in the official Azure documentation:

•  Create a Logic App in the Azure portal.
•  Add the HTTP connector and configure it to call the REST API endpoints.
•  Add the OAuth2.0 connector and configure it to authenticate using the Authorization Code grant type.
•  Add the Azure Blob Storage connector and configure it to store the data in Blob Storage.
•  Use the Control Actions to implement parallel execution of tasks.

Here are some links to the relevant Azure documentation that can help you get started with Azure Logic Apps:
https://techcommunity.microsoft.com/t5/azure-integration-services-blog/authenticate-standard-logic-apps-using-oauth-2-0-bearer-token/ba-p/3269576
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-azureblobstorage?tabs=consumption

I hope this information helps you. Let me know if you have any further questions or concerns.