How can I get notification that Event grid has ingested data from a csv/parquet file into ADX table?

Bexy Morgan 260

Scenario:

In a blob container csv/parquet file lands into the folder at regular intervals of time, as soon as files land into the container event grid ingests that csv/parquet file data to the table in ADX database.

How can I get the notification that " event grid has completed ingestion of 'fileName.csv' into ADX table"?

PRADEEPCHEEKATLA-MSFT 88,471 Reputation points Microsoft Employee

2023-09-04T07:40:09.6+00:00

@Bexy Morgan - Just checking in to see if the below answer provided by @Sander van de Velde helped, if this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA-MSFT 88,471 Reputation points Microsoft Employee

2023-09-11T08:28:34.4466667+00:00

@Bexy Morgan - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Bexy Morgan 260 Reputation points

2023-09-11T14:11:43.03+00:00

Hi @PRADEEPCHEEKATLA-MSFT , the provided response is not helpful

1 answer

Sander van de Velde | MVP 31,931 Reputation points MVP

2023-08-30T15:48:15.72+00:00

Hello @Bexy Morgan,

Azure Data Explorer shows ingestions using:

.show commands | top 100 by StartedOn

Ingestion is done in extents.

I wrote ingestion logic using an Azure Function and tagged new extends with the file name and more if needed. This way, I can prevent ingesting the same file twice and count the number of rows per file (please add the related).

Check out the 'bonus 2: tags' section here for an example of how to use the IngestBy / DropBy tags. The same goes for the creation time.

Back to your question, technically, this could be done with EventGrid ingestion ingestion properties too:

I have not done this myself, but...

... you can specify ingestion properties of the blob ingestion via the blob metadata.

Please let us know if it works for you.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Bexy Morgan 260 Reputation points

2023-08-31T08:58:10.11+00:00

Hi @Sander van de Velde | MVP , but with ingestion properties we could not get any confirmation about the "fileName" from which the data got ingested into ADX table using event grid. Please confirm me if I am wrong.

Also, can you please share your Azure function logic? Does your logic takes the "fileName" input as extent tag dynamically from the folder in ADX?

Sander van de Velde | MVP 31,931 Reputation points MVP

2023-08-31T10:24:07.91+00:00

Hello Bexy,

good question regarding how to get the filename using the regular data connection. The documentation is referencing adding blob metadata which you can edit here to test:

It seems tags are separated using commas.

Regarding the Azure function, the name is available this way:

public static void BlobTriggeredFunction([BlobTrigger("syncingest/{name}", Connection = "ImportStorage_Blob")] Stream myBlob, string name, ILogger log)
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How can I get notification that Event grid has ingested data from a csv/parquet file into ADX table?

1 answer

Your answer