SharePoint batch REST API request creates duplicate items. Is this a known issue? How to overcome this?

Question

SharePoint batch REST API request creates duplicate items. Is this a known issue? How to overcome this?

Deval 31 Microsoft Employee

I am using SharePoint batch rest APIs to create items in a SharePoint online list site.

The use case is this: There are more than 7000 items in source (which is not SPO) and they get updated/added/deleted frequently.

I am fetching these items or data from my source and adding them to one list in an SPO site and then using this data to create Power BI reports.

I am using batch REST APIs and first deleting all items from the list and then adding the new data in a batch of 100 items to the list.

I am using Power Automate with recurrence trigger to refresh the target or to perform above actions 4 times a day at regular intervals.

The issue is sometimes it works fine and creates exact no. of items, but sometimes it creates duplicates of items in all the batches.

I am sure that the duplicates are created by the batch request only because for all the duplicates the created time is exactly same in the SharePoint list. (see the exported data image)

Exported data

I am not sure what could be the issue, as no error is returned from the batch API response.

I am sure that this issue is not with the Power Automate because it is sending exact no of batches in both scenarios when this works fine and when it does not.

Is this a known issue with batch APIs?

What could be the possible solution for this?

1 answer

Your answer

Answer 1

RaytheonXie_MSFT 40,496 Microsoft External Staff

Hi @Deval,

In my experience, RESTful services with lots of traffic are bound to receive duplicate incoming POST requests unintentionally - e.g. sometimes a user will click 'Signup' and two requests will be sent simultaneously. This should be expected and handled by your backend service.

This issue could be handled in the backend using unique checks. You can add a GUID column and enforce unique in SharePoint list. When the duplicate request ocurs, the enforce unique will prevent duplicate item create.

Here is a similar issue with yours, please make a reference

https://stackoverflow.com/questions/15159274/avoid-duplicate-posts-with-rest

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

RaytheonXie_MSFT 40,496 Reputation points Microsoft External Staff

2023-03-02T05:44:05.26+00:00

Hi @Deval,

Have you tried the solution I proposed?

If you have any questions or progress, you can contact me in time.

Looking forward to your reply

Have a lucky day!

Thanks,

Raytheon Xie
Deval 31 Reputation points Microsoft Employee

2023-03-02T07:42:00.9333333+00:00
Hi @RaytheonXie_MSFT Thanks a lot for your response and sorry for the delayed reply.

Yes I have tried the suggestion of setting up "enforce unique value" for one column in the list and currently monitoring the solution.

But if you can provide some insights on the below queries related SharePoint batch REST APIs, that would be really helpful:

Even if I submit the batch requests in sequential actions of my Power automate, they are not executing in exact same sequence, they execute in asynchronous manner, is it expected?

I am basically using send http request to SharePoint action for submitting the batch requests. Now I get the response for these requests in sometime, but if I observe in the list, the item creation or deletion happens in async and even after the getting the response, the backend process of add and delete continues, is this also expected behaviour?
RaytheonXie_MSFT 40,496 Reputation points Microsoft External Staff

2023-03-03T08:11:34.1033333+00:00

Hi @Deval,

Since I am not familiar with Power automate. Microsoft has a dedicated forum to discuss about Power automate. You can get some more professional advice in the following forum.

https://techcommunity.microsoft.com/t5/power-apps-power-automate/bd-p/PowerAppFlow
RaytheonXie_MSFT 40,496 Reputation points Microsoft External Staff

2023-03-06T06:32:30.82+00:00

Hi @Deval,

Would you tell me whether your issue has been resolved or have any update?

I am looking forward to your reply.

Have a nice day!

Thanks,

Raytheon Xie
Deval 31 Reputation points Microsoft Employee

2023-03-06T06:50:19.2133333+00:00

@RaytheonXie_MSFT - I am 100% sure that this issue is not related to Power Automate as stated earlier in my details of the problem, the flow runs successfully end to end. The issue is with execution of SharePoint REST endpoint batch API requests from the backend, which are skipping some batches sometimes, sometimes executes the batches twice and executing them asynchronously. This behaviour is not mentioned in official documentation of batch endpoint

As far as flow is concerned, it is successfully running end to end and even in http requests, no errors are shown. Hence I do not want to post it on Power Automate forum and create ambiguity.

Clearly, here I need help/details from SPO backend perspective on why the batch requests have the above issues.

Thank you for your support and help here. It would be really helpful if there is some clarity in above stated issues. Many TIA.
Mateusz Radziszewski 0 Reputation points

2024-01-28T16:56:12.48+00:00

I have the same problem using Batch insert on D365 API. It seems then it is somewhere in MS ODATA backend where this issue is. I used the same approach as suggested here with additional GUID set up as unique index on the D365 level. But the issue is there and MS should check this out. I do not buy the stories like "you should check your backend as maybe you are firing requests twice or so", as this happens only on $batch endpoint. It never happens on single element create endpoints.

Share via

SharePoint batch REST API request creates duplicate items. Is this a known issue? How to overcome this?

1 answer

Your answer