I have an internal ticket tracking this issue through our subscription. In short, the data is generated on ADF side but not showing up on Log analytics. This has been escalated and tracked through internal ticket.
Logs are missing in the Log Analytics Workspace - Azure Data Factory starting 10/23/2020
Hello,
Our ADF and the log analytic workspace are hosted under West US2, same as our ADF, and since 10/23, we’ve noticed several pipeline runs do not come with the Succeeded or Failed log, but just InProgress or Queued. It threw off our query since we are trying to find the pipeline run time longer than usual.
My log retention is 30 days, and we have pay as we go plan. The log is under 5G in the last 90 days so it's relatively small.
Here is an example of the query by RunId that show the result only status in InProgress or Queued, no record of the same RunId with status failed or succeeded entry logged
Selected query results from Log analytics WS query -
But on our ADF we saw the following for the same RunId – None of them is still InProgress.
What could be the potential reason that the succeeded or failed entry is missing from the log analytics workspace?
Btw we’ve seen a 1-2 cases before, but we started to see a lot of more since 10/23, and it's still ongoing.