Hi, currently I 'have setup one materialized view, but I want to update view on at end of the day. Currently it's getting updated in near time. Is it possible in materialized view to schedule query on daily basis only ?

Hello @Khushbu Kella, welcome to this moderated Azure community forum. Materialized views expose an aggregation query over a source table or another materialized view. This article covers common and advanced use cases for materialized views. See also this documentation . These views are automatically updated when data in tables related to the view is ingested. Azure Data Explorer does not contain a scheduler. If you want to schedule tasks in Azure Data Explorer, you need to run some logic outside Azure Data Explorer eg. An Azure Function (using an ADX SDK ) or a Logic app . Then, you can change the source data which triggers the update of your materialized view. If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

How can I schedule materialized view (refresh daily data at EOD )in data explorer?

Accepted answer

Sander van de Velde 28,626 Reputation points MVP

2024-04-04T20:38:30.3166667+00:00

Hello @Khushbu Kella,

welcome to this moderated Azure community forum.

Materialized views expose an aggregation query over a source table or another materialized view. This article covers common and advanced use cases for materialized views.

See also this documentation.

These views are automatically updated when data in tables related to the view is ingested.

Azure Data Explorer does not contain a scheduler.

If you want to schedule tasks in Azure Data Explorer, you need to run some logic outside Azure Data Explorer eg. An Azure Function (using an ADX SDK) or a Logic app.

Then, you can change the source data which triggers the update of your materialized view.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.

1 person found this answer helpful.
Khushbu Kella 20 Reputation points Microsoft Employee

2024-04-05T05:39:20.4233333+00:00

Hi @Sander van Gennip Thanks for the reply.

I understand that Azure Data Explorer doesn't contain scheduler. I can schedule this query from other tools.

But the issue I'm facing is, source table (view will be on top of this), ingestion rate is very high. And I don't want to update same the view at the same rate. Daily update will also work for me.

So how can i stop or change the background materialized process frequency?

I see there is one batching policy, but threshold is too low for my use case.

So I wanted to know is there any alternative. Or any information on how background view process will impact source DB .

Thanks,
Khushbu

Sander van de Velde 28,626 Reputation points MVP

2024-04-05T15:00:01.58+00:00

Hello @Khushbu Kella,

so, you want to lower the compute on the cluster due to many updates on the source table.

You first can try to lower the batching frequency:

This will limit the amount of updates but also means the users have to wait longer...

You can also chose just using a regular function as a regular view. This way users have to wait (a bit) longer on the response but the whole CPU load could be less.

It depends on the number of calls expected and the number of updates to be performed.

Please try to measure what the actual impact is.
Sign in to comment

How can I schedule materialized view (refresh daily data at EOD )in data explorer?

0 additional answers