Azure Function for ETL

Question

Azure Function for ETL

Osama Ahmed 131

Hello!

I am fairly new to Azure Functions, and I am having some implementation problems. I am trying to perform an ETL process, where I get data from a REST API and then store this data into Azure Data Lake Storage. I must run this process twice per day so I know that I will be using a time-triggered Azure Function. Therefore:

Function Trigger = Timer Trigger

But I don't really understand the part of binding, will the REST API be an input binding while the ADLS Gen 2 the output binding?

Furthermore, when the function runs the first time, it will do a full data load to the ADLS, so does azure functions support incremental loads? (Future executions must ingest only new data, and NOT everything from the API again.

Accepted answer

0 additional answers

Your answer

Answer 1

MughundhanRaveendran-MSFT 12,506

Hi @Osama Ahmed ,

Thanks for reaching out to Q&A.

For your scenario, a timer trigger can be used as you mentioned. However, there is no inbuilt binding available for Data lake storage. Your understanding of the input and output binding is correct though.

Supported bindings in Azure functions : https://learn.microsoft.com/en-us/azure/azure-functions/functions-triggers-bindings?tabs=csharp#supported-bindings

Assuming that you are using .net as the language, you can make Rest API call using Http client and then write data to data lake storage using the Azure.Storage.Files.DataLake nuget library

https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-dotnet

Hope this helps!

Osama Ahmed 131 Reputation points

2022-11-29T13:06:52.313+00:00

Thank you @MughundhanRaveendran-MSFT for your answer. I will be using Python as the language as I am more familiar with it. Can the same approach your shared work with Python?
MughundhanRaveendran-MSFT 12,506 Reputation points

2022-12-01T05:05:29.503+00:00

@Osama Ahmed ,
Yes you can use the same approach in python as well, You can use http module in python to make rest api calls and use Azure Data Lake Storage client library for Python
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-python
Osama Ahmed 131 Reputation points

2022-12-01T08:57:34.263+00:00

Ok ok. Thank you very much
Osama Ahmed 131 Reputation points

2022-12-02T09:21:18.373+00:00

I have studied the link your provide and had a clear understanding. I would also like to inquire about storing the data in ADLS gen 2, everytime I call the API, I GET all the data available. I want to store it as following:

Container: Table_1
Directotry: Year/Month/Date
The full path as an example: Table_1/2022/December/{dd}

The azure function objective is to run daily and then store the daily data into the ADLS. I would appercaite if you can share any input on the implementation of this approach.

Share via

Azure Function for ETL

0 additional answers

Your answer