Azure Data Factory - Load web activity output into a SQL table

Question

I have some data in my SQL table. I have to pass the data in certain columns to a webservice and capture the output of the webservice in some other columns of the same table (Basically the webservice is designed to cleanse my data). I tried using lookup activity and pass the output of the columns in the body of the http request as dynamic content, but it does fetch only 5k rows. Did anybody face such scenarios before ?

Answer 1

Hello @Madhu Balan ,
Thanks for the ask and using the Microsoft Q&A platform .
At this time 5K is the limit for the Lookup(LU) activity and so you are hitting the limit . SInce your source is SQL you use the watermark logic if possible to get around this limit . The watermark is implemented here in this pipeline .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

You have to create a copy activity where you create HTTP Linked Service for the source.

The Sink can be a staging table in the SQL database, and you can create a store procedure for updating existing table.

It is also possible to use Stored Procedure for Sink table.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members