Ingestion of file from Share point to ADLS Gen2

Question

Hi Team,

I want to ingest the excel file from the shared point location to the ADLS Gen2 storage account.

Do we need to follow the steps from registration, as I can see the share point list connector in ADF? Please let me know the steps required for the same, from start to finish. I cant go with other Azure services so I need to do this via ADF only.

Thanks in advance.

Answer 1

Plz follow the below blog that provides step by step detail for the same :

https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/

The sharepoint list connector is to copy data from a sharepoint list and not files from sharepoint.

So you would have to use the REST API route of sharepoint

Answer 2

Hello Vrushabh Malbari,

Thank you for posting your query here!

Adding on to the previous response, you can copy files from SharePoint Online by using Web activity to authenticate and grab access token from SPO, then passing to subsequent Copy activity to copy data with HTTP connector as source.

https://learn.microsoft.com/en-us/answers/questions/53586/copy-files-from-sharepoint-into-azure-data-lake-st

Step by step using ADF:

If you haven't already, you need to create an Azure Data Factory instance in your Azure subscription. This can be done through the Azure Portal.

You need to create linked services in ADF for both SharePoint Online and Azure Data Lake Storage Gen2:

· For SharePoint Online: You'll need to create a linked service connecting to your SharePoint Online site. You would typically use the Office 365 authentication method, requiring your SharePoint URL, Authentication method (Office 365, Windows, or Anonymous), and your credentials.

· For Azure Data Lake Storage Gen2: Create a linked service for ADLS Gen2 using your storage account name, URL, and authentication method (such as account key, service principal, or managed identity)

After setting up the linked services, you need to create datasets that reference the linked services.

· For SharePoint: Create a dataset for the SharePoint folder you wish to copy. You need to specify the site URL and the folder path within your SharePoint site.

· For ADLS Gen2: Create a dataset for your ADLS Gen2 filesystem where you want to copy the SharePoint files and folders.

Next, you need to create a pipeline that defines the data movement and transformation activities.

· Copy Activity: Add a Copy activity to your pipeline. This activity will be responsible for copying data from your source (SharePoint) to your destination (ADLS Gen2) In the source settings of the Copy activity, select your SharePoint dataset In the sink settings, select your ADLS Gen2 dataset Configure the Copy activity to recursively copy files if you want to include subfolders and nested files.

Similar query that you may refer: https://learn.microsoft.com/en-us/answers/questions/1605103/is-there-any-way-to-copy-the-entire-folder-as-it-i

Do let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.