ADF dataflow timeout at Kusto Sink

Question

Hi all, I have created an Azure Data Factory dataflow where the sink is an ADX Kusto table. The dataflow does three joins in parallel and then unions the join result and writes to the sink. There're about 33 million rows written to the sink. And I'm having the following timeout error:
Operation on target Copy TechProfileAPICombined Primary failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'TechProfileUserSubscriptionPrimary': Error in post request:Connect to ingest-kustocxpcoprod.westus.kusto.windows.net:443 [ingest-kustocxpcoprod.westus.kusto.windows.net/52.250.220.31] failed: Connection timed out (Connection timed out)","Details":"shaded.msdataflow.com.microsoft.azure.kusto.data.exceptions.DataClientException: Error in post request:Connect to ingest-kustocxpcoprod.westus.kusto.windows.net:443 [ingest-kustocxpcoprod.westus.kusto.windows.net/52.250.220.31] failed: Connection timed out (Connection timed out)

I'm not sure if it's because the join is taking too much time due to the big size of the table. I have verified the connection is good. The pipeline can succeed when the dataset is small. I have tried scale out the ADX instance, increase the Kusto query timeout limit, and increase the dataflow compute size (to 128(+16 Driver cores)) but none of the approach works.

MicrosoftTeams-image (13).pngMicrosoftTeams-image (14).png

Answer 1

Hello @Zixuan Li,

Azure Data Explorer ingests data using either a batching or streaming mechanism.

Here, batching is used.

According to the documentation, batches are limited to a maximum size of 6GB.

The data size limit for a batch ingestion command is 6 GB.

It is recommended to add some kind of splitting mechanism in your data flow to prevent the batches from becoming too large.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.