Incremental Copy of Sharepoint files to adls

Prapul Kumar Dongari 116 Reputation points
2023-11-20T06:35:48.21+00:00

Hi team im facing an issue to get only newly inserted or modified files from sharepoint to adls what should i use in relative url as a query paramter to filter out latest files please give sujjestions.

Thanks

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,425 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,673 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,149 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Vinodh247 13,301 Reputation points
    2023-11-20T06:55:21.45+00:00

    HI Prapul Kumar Dongari:

    Thanks for reaching out to Microsoft Q&A.

    Yes, you can do by taking the modified date and time of the files through ADF pipeline easily. I have written an article with real-time demo, steps & screenshots in the link below.

    https://vinsdata.wordpress.com/2021/10/25/incremental-file-copy-in-azure-data-factory/

    Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.


  2. AnnuKumari-MSFT 32,011 Reputation points Microsoft Employee
    2023-11-20T09:14:11.8133333+00:00

    Hi Prapul Kumar Dongari ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

    From the description of your query, it seems you want to get only newly inserted or modified files from SharePoint to ADLS.

    When making a request to the SharePoint REST API, kindly try using the $filter query parameter to filter files based on their last modified date.

    The Site URL would look like below:

    https://<your-sharepoint-site>/_api/web/lists/getbytitle('<list-title>')/items?$filter=Modified gt datetime'2023-01-01T00:00:00'

    For more details, kindly visit the below documentation:

    Use Filter in share point REST API

    https://www.odata.org/documentation/odata-version-2-0/uri-conventions/#FilterSystemQueryOption

    Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

    0 comments No comments