Cannot save value for Batch Size in ADF Dataflow REST Sink

Oleksandr K 0

Every time I set Batch Size value (e.g. 100) in ADF Dataflow REST Sink it is always swept out after page refresh (or tab close/open) and it does not have effect if I Debug the Pipeline (run in debug mode).

Even if I commit manually batchSize in the Dataflow's code repository, when I open ADF UI and trigger Pipeline Debug, I always get an error that data records are sent to Sink one-by-one and not in batches.

phemanth 11,975 Reputation points Microsoft Vendor

2024-11-28T13:54:07.63+00:00
@Oleksandr K

Thanks for reaching out to Microsoft Q&A.

It sounds like you’re experiencing issue with the batch size setting in Azure Data Factory (ADF) Dataflow REST Sink.

Here are a few steps to resolve it:

Check Data Type: Ensure that the batch size value is set as an integer. Sometimes, the UI might expect a string, but the backend requires an integer.

Immediate Debug: After setting the batch size, try running the pipeline in debug mode immediately without refreshing the page. This has worked for some users.

Manual Code Commit: If you manually commit the batch size in the Dataflow’s code repository, make sure to verify the changes in the UI before running the pipeline.

Update ADF Version: Ensure that you are using the latest version of ADF, as updates often include bug fixes.

Hope this helps. Do let us know if you any further queries.
phemanth 11,975 Reputation points Microsoft Vendor

2024-11-29T10:06:16.8466667+00:00

@Oleksandr K We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Oleksandr K 0 Reputation points

2024-12-02T13:48:16.35+00:00

Hello @phemanth ,

1.Check Data Type: Ensure that the batch size value is set as an integer. Sometimes, the UI might expect a string, but the backend requires an integer.

Exactly. UI expects String, but backend expects Integer.

2.Immediate Debug: After setting the batch size, try running the pipeline in debug mode immediately without refreshing the page. This has worked for some users.

This sounds like a dirty workaround. Can you submit a bug for this? I doubt this is an expected behaviour.

3.Manual Code Commit: If you manually commit the batch size in the Dataflow’s code repository, make sure to verify the changes in the UI before running the pipeline.

When I set the value in code repository, then it is not displayed on UI.

4.Update ADF Version: Ensure that you are using the latest version of ADF, as updates often include bug fixes.

Please advise how can I do this. I can see that we are using Data factory (V2)
phemanth 11,975 Reputation points Microsoft Vendor

2024-12-03T21:06:35.43+00:00

@Oleksandr K We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Share via

Cannot save value for Batch Size in ADF Dataflow REST Sink

Your answer