Logs are missing in the Log Analytics Workspace - Azure Data Factory starting 10/23/2020

Helen Ho 1 Reputation point Microsoft Employee
2020-11-05T18:59:56.11+00:00

Hello,

Our ADF and the log analytic workspace are hosted under West US2, same as our ADF, and since 10/23, we’ve noticed several pipeline runs do not come with the Succeeded or Failed log, but just InProgress or Queued. It threw off our query since we are trying to find the pipeline run time longer than usual.

My log retention is 30 days, and we have pay as we go plan. The log is under 5G in the last 90 days so it's relatively small.

Here is an example of the query by RunId that show the result only status in InProgress or Queued, no record of the same RunId with status failed or succeeded entry logged

Selected query results from Log analytics WS query -
37708-la.jpg

But on our ADF we saw the following for the same RunId – None of them is still InProgress.

37821-fail.png

37831-pass.png

What could be the potential reason that the succeeded or failed entry is missing from the log analytics workspace?

Btw we’ve seen a 1-2 cases before, but we started to see a lot of more since 10/23, and it's still ongoing.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Helen Ho 1 Reputation point Microsoft Employee
    2020-11-16T22:28:52.45+00:00

    I have an internal ticket tracking this issue through our subscription. In short, the data is generated on ADF side but not showing up on Log analytics. This has been escalated and tracked through internal ticket.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.