How can I monitor and identify ADX ingestion fails at file level by using Queued file ingestion type with SDK library ?

Question

Hi community,

I have implemented ADX Queued ingestion ( QueueIngestionLink ) to ingest timeseries data passed by csv files. The java application is implemented by using SDK library as suggested from Microsoft.
I have left default ingestion policy as I showed below:

To understand better the Queued ingestion file behavior, I have tried to ingest a file per time to see the result on my Telemetry table. Being only a file for testing the policy batch was triggered by Time.

How I could see from the Ingestion Time field of the Telemetry table I have seen 4 batches for 4 different files. As I'm supposing in other situations we could have more files in a single batch.

I understood that this approach is asynchronous attributable to Producer and Consumer pattern, but in case of the batch ingestion failed, is there a way identify the file or files have caused the failure ?

In nutshell is there the possibility to know failure at file level ?

Otherwise do you have the suggestions ?

Answer 1

Hello @Stefano,

the queued ingestion is following the Batching rules by default.

The ingestion time is available via this Kusto statement:

T | extend ingestionTime = ingestion_time() | top 10 by ingestionTime

You can check ingestion using:

.show commands | top 100 by StartedOn

You can check the ingestion failures with

.show ingestion failures

There are multiple ways to check the behavior of ingestion. See also this blog post regarding the 'Drop by' tags and 'Ingest By' tags on extents, the ReportLevel and Report Method ingestion properties, and altering the ingestion time.

The batching strategy can be overridden completely when the streaming ingestion policy is overruled.

I recommend investing some time to play with these settings. This video gives you some pointers too.

---

This answer is written by myself, not by some AI. If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.