Can we push data from ADLS GEN 2 / SQL POOL to rest api end point using Azure synapse pipeline? If Yes, Is there any document to follow for this process?

Bhanu 1

We have a requirement to create a synapse pipeline to push the csv file data from ADLS GEN2/azure sql pool to REST API end point.

MartinJaffer-MSFT 26,061 Reputation points

2022-11-03T15:53:03.963+00:00

Hello @Bhanu and welcome to Microsoft Q&A

As I understand, you wish to copy data to a REST endpoint. The rest connector now can be used a sink. The specifics depends upon your REST service. They can work in different ways. The rest connector will send data in JSON format. Without knowing more about your desired sink, I cannot tell you whether to use POST or PUT or PATCH.

ADLS Gen2 and dedicated SQL pool are very different, but both can be used as source in copy activity. That would be the easy part.
MartinJaffer-MSFT 26,061 Reputation points

2022-11-08T16:48:50.85+00:00

Hello @Bhanu ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .

Thanks
Martin

1 answer

Bhanu 1 Reputation point

2022-11-10T21:52:15.083+00:00

Heloo @MartinJaffer-MSFT ,
Thank you for your response. I have used the rest connector now can be used a sink which you have shared in the first comment.
I have used the we activity function in synapses to read data from the adls container and post the data to REST endpoin.

QQ: I have the requirement to post csv file which should be in BASE64 format and then push to REST API end point.
Does the REST as SINK only accepts json format? or can we convert csv to base64 and the post to REST API endpoint using WEB ACTIVITY.

Thanks in advance!
Bhanu
Please sign in to rate this answer.
MartinJaffer-MSFT 26,061 Reputation points

2022-11-10T23:57:37.41+00:00

Hey @Bhanu I was wondering how REST as sink worked myself, I hadn't cause for it til now so I did a test.

I used copy activity with source as a csv blob, and the sink as REST pointing at a blank logic app triggered by http. After doing a debug run, we looked at the logs for the logic app, and found that the Factory turned the csv into JSON.

So looks like using the Web activity as you are is the only way right now. It is good you are using the base64. Once I tried to do something similar, with compressed data in web activity, and that somehow got corrupted. At that time I didn't understand about base64.
Sign in to comment