Incremental Copy of Sharepoint files to adls

Question

Incremental Copy of Sharepoint files to adls

Prapul Kumar Dongari 116

Hi team im facing an issue to get only newly inserted or modified files from sharepoint to adls what should i use in relative url as a query paramter to filter out latest files please give sujjestions.

Thanks

Vinodh247 33,391 Reputation points MVP Moderator

2023-11-22T06:16:01.7566667+00:00

Just checking if you had a chance to look at any of the answer that worked for you. If so, can you please upvote and accept that as an answer?
AnnuKumari-MSFT 34,546 Reputation points Microsoft Employee Moderator

2023-11-28T07:17:01.9166667+00:00

Hi Prapul Kumar Dongari ,

Just following up to see if the below answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'

2 answers

Your answer

Vinodh247 33,391 Reputation points MVP Moderator

2023-11-22T06:16:01.7566667+00:00

Just checking if you had a chance to look at any of the answer that worked for you. If so, can you please upvote and accept that as an answer?
AnnuKumari-MSFT 34,546 Reputation points Microsoft Employee Moderator

2023-11-28T07:17:01.9166667+00:00

Hi Prapul Kumar Dongari ,

Just following up to see if the below answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'

Answer 1

Vinodh247 33,391 MVP Moderator

HI Prapul Kumar Dongari:

Thanks for reaching out to Microsoft Q&A.

Yes, you can do by taking the modified date and time of the files through ADF pipeline easily. I have written an article with real-time demo, steps & screenshots in the link below.

https://vinsdata.wordpress.com/2021/10/25/incremental-file-copy-in-azure-data-factory/

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Prapul Kumar Dongari 116 Reputation points

2023-11-20T07:01:42.7766667+00:00

Hi @Vinodh247 thanks for your reply but the sharepoint apporach is different from file system apporach here we using HTTP connector to bring files from sharepoint so my query is how can i filter files based on lastmodified date in URL.
Vinodh247 33,391 Reputation points MVP Moderator

2023-11-20T07:40:58.6433333+00:00

Oh SharePoint! Sorry I missed that.

Easy solution would be to isolate those files in a separate folder on the source system itself.

But I am not sure if this is possible at all.

Let me check if there are any other solution.

Answer 2

Hi Prapul Kumar Dongari ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

From the description of your query, it seems you want to get only newly inserted or modified files from SharePoint to ADLS.

When making a request to the SharePoint REST API, kindly try using the $filter query parameter to filter files based on their last modified date.

The Site URL would look like below:

https://<your-sharepoint-site>/_api/web/lists/getbytitle('<list-title>')/items?$filter=Modified gt datetime'2023-01-01T00:00:00'

For more details, kindly visit the below documentation:

Use Filter in share point REST API

https://www.odata.org/documentation/odata-version-2-0/uri-conventions/#FilterSystemQueryOption

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Share via

Incremental Copy of Sharepoint files to adls

2 answers

Your answer