This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 90%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA0%3C/text%3E%3C/svg%3E)
We came across a situation where certain logs are missing of in the ADFPipelineRun in the log analytics workspace.
For instance a log of Succeeded pipeline execution is missing even if the pipeline has finished the execution successfully. and there are several such
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
Hi @Anonymous ,
Welcome to Microsoft Q&A Platform.
Sorry that you are experiencing this issue. There is a known limitation of 500 column limit in Azure monitor logs and AzureDiagnostics table is in particular susceptible to this limit since it includes properties for all Azure services writing to it. Once this limit is reached, any rows containing data with any property outside of the first 500 will be dropped at ingestion time. Possible workarounds are given in the below document as well. Please let us know if your scenario is different and also more details - Pipeline Run ID that is missing and region of the data factory for us to investigate further.
Lets take example of Run id : 35f19e81-92b7-433e-86ea-5be4578472f8
Its seen in progress in Azure Logs and never succeeded.
Similar is the case with 1924241c-4665-448f-83a3-3470ea1dfe4f and many more, however there are many where all 3 entries are logged i.e. Queued, In Progress and Succeeded
We rely on these status to send out alerts if a Job is In Progress for more than 15 minutes and a missing Success Entry for actually a successful run is generating false alerts
Hi @Anonymous ,
This is strange. We are working closely with the internal teams to understand the root cause of this. I will get back to you as soon as I hear update from them.
Thanks for your patience!
Hi @Anonymous ,
This is to update that I am still working with the internal teams to investigate the issue. Can you please suggest if you are using "Azure Diagnostics" or "Resource Specific" settings in diagnostics settings and workspace id that will help further investigation.
Thanks for your patience!
Hi @HarithaMaddi-MSFT
Its Resource Specific
Thanks @Anonymous for clarifying. As mentioned above, can you also please share your workspace id for further investigation.
Here it @HarithaMaddi-MSFT
9eaea0cf-8523-4b34-b1cc-6213db5f2f2c
Thanks @Anonymous . I shared these details with internal team and will get back to you as soon as I get more insights.
Thanks for your patience again!
Hi @Anonymous ,
Wanted to share an update that I am still working with multiple internal teams to understand the root cause of it. Thanks for your patience!
Hi @Anonymous ,
Thanks for your patience.
Our internal team investigated and observed that this was caused due to outage in WestUS region where certain logs were accidentally deleted and it was confirmed that the issue is fixed now.
Please let us know if issue reoccurs and we will further investigate it with our team.
Hi @Anonymous ,
We have not received a response from you. Please let us know if you are still facing the issue, we will continue to engage with you.
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Hi @Anonymous ,
We still have not heard back from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
I am sorry for replying late. Let me monitor this for a while and if this happens again i'll report it back
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 9%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Hello,
I can confirm this issue has not been fixed:
I've got multiple runs where I'm missing logs for "Queued" and "InProgress" status.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 51%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECP%3C/text%3E%3C/svg%3E)
I am seeing multiple issues where the Succeed records are not showing in log analytics, but the InProgress and Queued records are showing. I can actually find the ActivityRunId on the pipeline in the ADF UI and it shows as Succeeded, but there is no Succeeded record in log analytics.
This is important as we're looking at getting detailed costs down to the activity and pipeline level, but with missing records it will never be completely accurate.