copy files in bulk from sharepoint to adls

Question

copy files in bulk from sharepoint to adls

T M, Chaitra SBOBNG-ITY/FB 6

hello All,
how to copy files from sharepoint in bulk to adls from azure data factory,c an you help with any links

Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-04-27T01:36:17.68+00:00

Hi @T M, Chaitra SBOBNG-ITY/FB ,

Thanks for using Microsoft Q&A !!

I do not think there is a direct way to copy file in bulk. You may try to use a web activity to call SharePoint REST API (something like _api/web/lists/getbytitle('Documents')/items) to the get list of file names on SharePoint and then go through the list of files (using ForEach) to copy files one by one by (using Copy activity as suggested below which copies a single file). I do not have a working sample for the same but you can try if this helps to achieve what you are looking for.

Thanks
Saurabh
MartinJaffer-MSFT 26,236 Reputation points

2021-05-04T17:16:14.383+00:00

@T M, Chaitra SBOBNG-ITY/FB if this issue has been resolved, please mark as accepted answer.

2 answers

Your answer

Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-04-27T01:36:17.68+00:00

Hi @T M, Chaitra SBOBNG-ITY/FB ,

Thanks for using Microsoft Q&A !!

I do not think there is a direct way to copy file in bulk. You may try to use a web activity to call SharePoint REST API (something like _api/web/lists/getbytitle('Documents')/items) to the get list of file names on SharePoint and then go through the list of files (using ForEach) to copy files one by one by (using Copy activity as suggested below which copies a single file). I do not have a working sample for the same but you can try if this helps to achieve what you are looking for.

Thanks
Saurabh
MartinJaffer-MSFT 26,236 Reputation points

2021-05-04T17:16:14.383+00:00

@T M, Chaitra SBOBNG-ITY/FB if this issue has been resolved, please mark as accepted answer.

Answer 1

T M, Chaitra SBOBNG-ITY/FB 6

Hello SaurabhSharma-msft and kasperulvedal,

Thanks for you time in addressing this.. it worked , and now one file is getting copied from sharepoint to ADLS
issue was:
1>Relative server url i missed a sub folder while fetching file
2> $value i used before now OpenBinaryStream i used , neverthless both of these would seem to give results

now my open question is , how would i dynamically iterate over files tat gets added in sharepoint,
GetMetaData wont work for https i believe

Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-04-30T18:10:10.17+00:00

@T M, Chaitra SBOBNG-ITY/FB Good to hear that it worked. Like mentioned earlier, you may need to check for another web activity to get the list of items using SharePoint Rest API (something like _api/web/lists/getbytitle('Documents')/items), though I haven't used this myself but could be a starting point.

Thanks
Saurabh

Answer 2

Kasper Ulvedal 11

Have a look at this article from the Docs: https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list#copy-file-from-sharepoint-online

Best,
Kasper

T M, Chaitra SBOBNG-ITY/FB 6 Reputation points

2021-04-27T10:27:49.563+00:00

getting the below error when followed the steps of the document
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-04-27T14:29:31.45+00:00

@T M, Chaitra SBOBNG-ITY/FB Please check if you are using the correct path to access the Sharepoint document. For example -
If your file resides under on SharePoint under Shared documents -

then you have to use the Base URL in the source as below in your HTTP linked service -
https://<sharepoint name>.sharepoint.com/_api/web/GetFileByServerRelativeUrl('/Shared Documents/Sample.xlsx')/$value

Please let me know if still see any issues.

Thanks
Saurabh
Kasper Ulvedal 11 Reputation points

2021-04-28T05:37:51.32+00:00
If HTTP returns 404, I think something is wrong with the URL you are calling on the source side. It is also what @Saurabh Sharma explains above.

Though, I also tried to set up a pipeline as described in the Docs. The /$value do not work with binary copy activities - I used /OpenBinaryStream instead, and it worked.

So:

Set the Sink to a binary dataset.

Use the following base URL as source: https://[site-url]/_api/web/GetFileByServerRelativeUrl('[relative-path-to-file]')/OpenBinaryStream

site-url: contoso

relative-path-to-file: /SharedDocuments/File.txt

Please let us know if this works :)

Share via

copy files in bulk from sharepoint to adls

2 answers

Your answer