How can I get notification that Event grid has ingested data from a csv/parquet file into ADX table?

Bexy Morgan 175 Reputation points


In a blob container csv/parquet file lands into the folder at regular intervals of time, as soon as files land into the container event grid ingests that csv/parquet file data to the table in ADX database.

How can I get the notification that " event grid has completed ingestion of 'fileName.csv' into ADX table"?

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
365 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Sander van de Velde 22,031 Reputation points MVP

    Hello @Bexy Morgan,

    Azure Data Explorer shows ingestions using:

    .show commands | top 100 by StartedOn

    Ingestion is done in extents.

    I wrote ingestion logic using an Azure Function and tagged new extends with the file name and more if needed. This way, I can prevent ingesting the same file twice and count the number of rows per file (please add the related).

    Check out the 'bonus 2: tags' section here for an example of how to use the IngestBy / DropBy tags. The same goes for the creation time.

    Back to your question, technically, this could be done with EventGrid ingestion ingestion properties too:

    User's image

    I have not done this myself, but...

    ... you can specify ingestion properties of the blob ingestion via the blob metadata.

    Please let us know if it works for you.

    If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.