How can I get all files inside a drive irrespective of folder structure in ADF?

Mansi Yadav 0 Reputation points
2024-05-14T00:43:50.9066667+00:00

I want to copy files files from sharepoint drive which has lots of nested folders. Maximum hierarchy of folders is for 12 levels. Currently I'm using below endpoint in ADF's Web activity as it was mentioned in some of the articles that it provides every element inside the drive at any folder level.

https://graph.microsoft.com/v1.0/drives/@{pipeline().parameters.DriveID}/items/?$filter=lastModifiedDateTime ge @{pipeline().parameters.lastUpdate}&$count=true

I was handling odata. next link also. But when I executed the pipeline, it was running fine but after 5th cycle I noticed that results were somewhat similar to the previous one. It copied only 600 files out of 50k files.

Please suggest me one endpoint that can easily fetch all the files or folders at different folder level.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,753 questions
{count} votes