How to debug if my DCR isn't working
I have an on prem Linux server that was onboarded successfully through Azure Agent. I verified the server's status through KQL -> Heartbeat
My purpose for this is to collect Custom application logs from this server and feed it to Log Analyitcs Workspace through DCR. I have followed everything here:
https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-custom-logs
I have successfully created the Log Analytics Workspace -> Custom table under this workspace
Created the DCR as well, connect the on prem server as resource for this DCR, add the file patterns and file location in the source as well.
But I can't see the rawdata when I try to query it in LogAnalytics Workspace -> Logs using
MyCustomTable_CL | project RawData
So how to debug if the Data Collection Rule is really working? or where is the issue in this?
Here is my Source config:
And here is what the log data looks like in my Linux server:
2025-01-28 08:04:44(PID = 79773) (HOST = 98yd98sadsaddsa.org) INFO - DB CONNECT 'dsdsada'
This is inside .log file
Azure Monitor
-
Naveena Patlolla • 400 Reputation points • Microsoft Vendor
Jan 29, 2025, 6:21 AM Thanks for posting your query and using Microsoft Q&A Forum,
We are checking with our internal team to get more information related to your query and will get back to you as soon as once we have an update.
-
Naveena Patlolla • 400 Reputation points • Microsoft Vendor
Jan 29, 2025, 1:11 PM Since you're not seeing the raw data when querying, can you please try the below troubleshooting steps:
Check Data Collection Status: Ensure that the data collection is active and not hitting any limits. You can run the following query in KQL to check the status:
Operation | where OperationCategory == 'Data Collection Status'
Look for OperationStatus values of Succeeded or Warning
Check Logs for Errors: Look at the DCR error logs for any ingestion errors. You can query the DCRLogErrors table to see if there are any logged errors related to your DCR. Azure Monitor Agent logs are stored in:
var/opt/microsoft/azuremonitoragent/log/mdsd.log
Verify the File Path in DCR
- Ensure the file path matches exactly what’s on the server.
- The path should be absolute and not relative.
var/log/mycustomapp*.log
- If you specified a folder, ensure all files inside match the pattern. Manually Test the DCR
- Please refer the below MS document: https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-custom-logs
let us know if you have any further queries. I’m happy to assist you further.
-
Naveena Patlolla • 400 Reputation points • Microsoft Vendor
Jan 30, 2025, 5:11 AM Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.
If the comment is helpful, please click "Upvote it"
Thankyou
-
Caday, Belmark June • 5 Reputation points
Jan 30, 2025, 5:32 AM Hi @Naveena Patlolla So what does it mean if :
Operation | where OperationCategory == 'Data Collection Status'
and
DCRLogErrors
doesn't return any result? even if I adjusted the timeframe. Using
Heartbeat
I can still see result proving that the resource (on-perm server that was on-boarded) is still connected via Azure Arc.
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 30, 2025, 9:29 AM Hello Caday, Belmark June, can you try checking the DCR is correctly associated to the Linux server that you are using! Also, verify if the agent is sending the logs to the server.
Check the logs with command:
sudo journalctl -u azuremonitor-agent
-
Caday, Belmark June • 5 Reputation points
Jan 30, 2025, 10:18 AM @Madugula Jahnavi "verify if the agent is sending the logs to the server" How?
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 30, 2025, 10:21 AM Visit your Linux server and try to execute the command I provided. Caday, Belmark June
-
Caday, Belmark June • 5 Reputation points
Jan 30, 2025, 11:03 AM The command doesn't return anything. Also I checked this log file mdsd.err, it has an interesting entry which i am unable to make sense:
Mcs Callback didn't result in a configuration download. Abort mdsd./etc/opt/microsoft/azuremonitoragent/config-cache/etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.lkg.json/etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 30, 2025, 11:49 AM This is strange error. Have you referred this https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm#issues-collecting-syslog for troubleshooting it! Caday, Belmark June
-
Caday, Belmark June • 5 Reputation points
Jan 30, 2025, 11:52 AM Yes, that's what I was reading before I checked the logs in my Linux server. That's how I learned to check the mdsd.* logs
So what does it mean? is it perhaps the cause why the agent is not collecting logs?
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 30, 2025, 11:57 AM Caday, Belmark June Ok understood. And also, if you still do not able to see the logs emitting in msd files it might be related to authentication issue. Have you added the syslog user to the group of himds as given clearly here in https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm#troubleshooting-issues-on-arc-enabled-server.
-
Caday, Belmark June • 5 Reputation points
Jan 30, 2025, 12:01 PM - I don't see any authentication related error in mdsd.err log file.
- Yes, syslog is already part of group himds, verified by executing the following command:
id syslog
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 30, 2025, 12:37 PM Have you checked the Linux server network settings whether they are blocked by any inbound or outbound internet connection, which is making the agent cannot collecting the custom logs. Caday, Belmark June
-
Caday, Belmark June • 5 Reputation points
Jan 30, 2025, 12:51 PM I successfully on-boarded the Linux machine Azure Arc, that means it has connectivity to Azure. Ping www.google.com is good too, so it's not internet connection issue. And again, I verified that the resource (my Linux machine) is connected to Azure, how? Via Heartbeat in KQL.
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 31, 2025, 3:29 AM Now is the issue resolved ? Caday, Belmark June
-
Caday, Belmark June • 5 Reputation points
Jan 31, 2025, 6:58 AM Bro @Madugula Jahnavi I just said it's not connectivity issue, c'mon.
-
Madugula Jahnavi • 0 Reputation points • Microsoft Vendor
Jan 31, 2025, 8:07 AM Caday, Belmark June Can you try enabling the DCR error logs for debugging purpose by referring to the MS Doc. https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-monitor#enable-dcr-error-logs
Sign in to comment