how to pass data from delta lake to rest api in azure

Question

how to pass data from delta lake to rest api in azure

Laxmi Kakarlapudi 21

Hi,

I want to build a pipeline where I take data from delta lake and post it to rest api and collect responses.
How do I pass delta lake data dynamically to rest api. Here is the activity I have done :

copy activity where rest is source and sink is adls gen2
It worked well. But I hard coded records in the body of the source. Now my question is how do I pass it dynamically from delta lake tables.

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

OmarSiado-MSFT 156 Microsoft Employee

Hi @Laxmi Kakarlapudi ,

I think maybe thinking on batch load schema could work for you, does your API also requires date to load only date related information?, if so, you could create a pipeline parameter and send to the API for example, then also using date for creating folder by date adding files there (here you can find different structures i think depends on your scenario).

Please let us know if that works for you or help us clarifying your scenario with more details so that we will be able to provide suggestions.

Regards,

OCS

Laxmi Kakarlapudi 21 Reputation points

2021-09-04T01:46:35.543+00:00
Hi Omar,

Thank you for your response.
Here is the design that I want to create.

copy data from delta lake to adls gen2

send the o/p from 1 as input to rest api(post) and collect responses.

Save the responses to adls gen2

I am stuck with couple of things here.
For step 1, when I create copy activity to copy data from delta lake to adls gen2 it fails with fs azure account key configuration issue. I did save access key and container in the cluster - advanced settings. But still it is not working.

For step 2, I need to take o/p from step 1 as a body in the rest api activity(post). How can I do that.

For step 3, how to save o/p from step 2 into storage account.
Laxmi Kakarlapudi 21 Reputation points

2021-09-04T02:23:17.547+00:00

Also I need to copy data from delta lake as a json into storage account so that I can pass json data as an input to rest api post activity. How to do that?
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2021-09-08T18:09:49.013+00:00

Hi @Laxmi Kakarlapudi ,

You can consider copy activity with Sink data type as json to store data in json format. After that use lookup activity to read data from that json and then pass output of lookup activity to rest api using web activity.
Laxmi Kakarlapudi 21 Reputation points

2021-09-09T14:34:28.19+00:00

Hi,

Thank you for all your responses.

I did try your suggestions and it worked. However it took longer time to process(14 hrs).
I think I need to use streaming. Do anyone have an idea or a video to follow.

Thank you.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2021-09-10T16:09:28.43+00:00

Hi @Laxmi Kakarlapudi ,

Thank you for confirming that suggestions worked for you. Could you please mark it as Accepted Answer? Accepted answers helps community as well. Thank you.
OmarSiado-MSFT 156 Reputation points Microsoft Employee

2021-09-10T16:57:38.757+00:00
Hi @Laxmi Kakarlapudi ,

Thanks for your reply, i think it really depends on the scenario you're facing, there're some variables that have not being considered within your specific question, if you want to do Web API Call then you could use ADF Web Activity component and send parameters for example by Body in JSON format, what i would not recommend is sending a huge amount of data this way, here's a link for documentation:
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

On the other hand if what you need is to apply some business rules based on two different data sources i would recommend is following the strategies bellow

Create staging/bronce area into datalake then create data flow and add the different data sources to apply business rules there.

Create staging area within database then apply business rules via stored proc to get final data into the DB, here you'll need to do stored procedure calls for example to execute your business rules.

Please let us know if that works for you, if so, could you please mark as Accepted Answer?, Accepted answers helps community as well as @ShaikMaheer-MSFT already mentioned.

Thanks!
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2021-09-28T10:20:21.867+00:00

Hi @Laxmi Kakarlapudi ,

Thank you for marking as accepted answer.

Share via

how to pass data from delta lake to rest api in azure

0 additional answers

Your answer