ETL logs in Azure Data Factory

Question

Hello,

I am trying to get some ETL logs in adf pipelines.

Firstly i need to get the exact duration of dataflow run.You can see below.

But when i check the logs,i cannot find the attribute in json logs.

I can find duration as coreHour unit which represent the billableDuration.

Is there any way to get these 11 and 13 seconds ?

Secondly,

I am trying to write this datalogs into a blob file,but i need to write them row by row.

For example,i will add this copy data1 (which include query of logs as you see below)

SELECT '@{pipeline().DataFactory}' as DataFactory_Name,
'@{pipeline().Pipeline}' as Pipeline_Name,
'@{pipeline().RunId}' as RunId,
'@{activity('STGJsn_Gl_Accounts_INACTIVE').output.effectiveIntegrationRuntime}' as IntegrationTuntime, 
'@{activity('STGJsn_Gl_Accounts_INACTIVE').output.runStatus.metrics.StgJsnGlAccounts.rowsWritten}' as rowsWritten

and after i run this for second time i still see 1 record which include only last runs logs.

Can i also arrange this cumulative ways ?

Thank you

Answer 1

Hello Mustafa Ispir

You can write a stored procedure which will perform the operations in a sql table if you want to append the data, I guess you are overwriting the file every time as the log file name is the same

In case of activity run duration, for instance for Copy activity you can pass the following to the sproc: @activity('COPY_ACTIVITY_NAME').output.copyDuration

This will pass the right duration to the sproc and it will perform append to the log table, you can use that log table in SQL to monitor the ETL logs, if you compulsorily need a file then you can write a custom code which will perform the append in the file (this will be kind of a transaction insert logic)

Answer 2

@Mustafa Ispir The duration you pointed out is not in the output, but in the activity itself.

@{activity('STGJsn_Gl_Accounts_INACTIVE').Duration}

not

@{activity('STGJsn_Gl_Accounts_INACTIVE').output.Duration}