How to know update policies are working fine in Kusto

ADX 156 Reputation points
2021-12-16T04:16:41.857+00:00

Hi ,

I have two tables mentioned below.

Stage table : stg_test (Columns: Id:int, Name:string, Location:string)
Final Table: fnl_test (Columns: Id:int, Name:string, Location:string)

stg_test table is always truncate and load on daily basis and this table is being loaded by ADF pipeline.

I have created the update policy when ever the data ingested into stg_test table the data will be updated in the fnl_test table.

Function: .create-or-alter function fntest {
let HistData = (fnl_test|HistId=Id);
stg_test
|join kind=leftouter (HistData) on $left.Id==$right.HistId
|where isnull(HistId)
|project Id, Name,Location
}

update policy statement: .alter-merge table fnl_test policy update @'[{"IsEnabled": true, "Source": "stg_test", "Query": "fntest()", "IsTransactional": true, "PropagateIngestionProperties": false}]';

The update policies are working fine and data is updating in fnl_test table whenever the data is inserted in stg_test table.

but my question is how do we know how many records are processed in fnl_test table from stg_test table for each time and if there are any data conversion issues the data wont get update in the fnl_test table in this case how do we know the failures. Is there any way we can capture the log information and load into into the Customized error log table. Appreciate your help on this.

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
508 questions
{count} votes