This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 94%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Hello there, I have a requirement to fetch the billions of records of data from the salesforce objects on daily basis, I am using the BULK API 2.0 for this task to get the data quickly as the salesforce direct connector is taking more than 15 hours to extract all the records from one object, so I got the bearer token first and created a query job using web activity and the job is processing all the records as expected and currently Salesforce BULK API is supporting CSV format only. so I have created http linked service and copying the data using the copy activity, since there are millions of records I am not able to fetch all at once, so we need to implement pagination here.
I have gone through the salesforce bulk api 2.0 documentation and found that if all the records not process in single request then http response body sends a locator value which we can pass as argument for the next set of results.
Here is the sample response from the salesforce documentation
HTTP/1.1 200 OK
...
Sforce-Locator: MTAwMDA
Sforce-NumberOfRecords: 50000
...
"Id","Name"
"005R0000000UyrWIAS","Jane Dunn"
"005R0000000GiwjIAC","George Wright"
"005R0000000GiwoIAC","Pat Wilson"
I have tried with the web activity and I can see the locator value and the response header details as below
"ADFWebActivityResponseHeaders": {
"Date": "Mon, 03 Jan 2022 07:33:06 GMT",
"Set-Cookie": "CookieConsentPolicy=0:1; path=/; expires=Tue, 03-Jan-2023 07:33:06 GMT; Max-Age=31536000;LSKey-c$CookieConsentPolicy=0:1; domain=; path=/; expires=Tue, 03-Jan-2023 07:33:06 GMT; Max-Age=31536000;BrowserId=ZDuTmmxnEeyppMMTrylLrg; domain=.salesforce.com; path=/; expires=Tue, 03-Jan-2023 07:33:06 GMT; Max-Age=31536000",
"Strict-Transport-Security": "max-age=31536000; includeSubDomains",
"X-Content-Type-Options": "nosniff",
"X-XSS-Protection": "1; mode=block",
"X-Robots-Tag": "none",
"Cache-Control": "no-store, must-revalidate, no-cache, max-age=0, private",
"Sforce-Limit-Info": "api-usage=42264/5000000",
"Sforce-Locator": "NTAw",
However I can't see the same header details in the http response currently to implement the pagination technique, can anyone support me here.
Plz Note: we can't use the REST API connector as the source supports only CSV format.
@ShaikMaheer-MSFT @KranthiPakala-MSFT @MartinJaffer-MSFT
can anyone of you check this and help me here.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Hello @Muralikrishna Pirla . I have begun looking into this difficulty. I did a quick test of my own to see what is available from the copy activity and did not see any headers either. So far, I see two possible work arounds.
Since you are able to get the info you need through web activity, we could use that, and pass the output body to another web activity which will write to blob/datalake.
Or we could use 2 calls to your api, one using web activity to retrieve the locator, the other using http copy activity to move the data. This is assuming, that the api will predictably return the same output for both calls.
Actually, I never asked, what are you using for a sink dataset?
In the longer-term we could put a feature request to make the headers available for the copy activity.
Hi @MartinJaffer-MSFT Thanks for the response.
I can't really use the web activity as it can't return more than 4mb data and I have to fetch around 10 millions of records daily, if I put it in the loop it will take 10 to 15 hours.
my destination is ADLS Gen1.
Just curios, can we use the logic apps here.
Yes @Muralikrishna Pirla , from a quick look, Logic Apps does appear to be an option. I am not well versed in Logic Apps, but it is probably easier to accomplish your goal there.
In checking for limitations I found:
Logic Apps does have a salesforce connector. You can also try using HTTP in case the connector doesn't do bulk. I think it does, I found mention of bulk under jobs/actions here.
For outbound HTTP requests, the timeout is either 120 or 235 seconds.
For message size limits, 100MB without chunking, or 1GB with chunking. The salesforce connector may be different.
Hi @MartinJaffer-MSFT we are able to read the HTTP Header values in the Azure Logic Apps, however logic apps are not supportive here with large datasets, we can't extract more than 5k records with the logic apps.