Custom logs for Linux Arc sever not coming to portal but are there on machine

Question

I have configured Custom logs with Azure arc linux server via dcr association but the logs are not coming in portal. I checked the path in linux arc machine, the logs are visible there but not coming to the portal. Can you please tell what is the issue?
Also the heartbeat of agent is coming.

Answer 1

Priya Goswami, thank you for additional details.

I am not sure what you meant by txt path - but I checked and the td-agent.conf file in my "working" scenario is identical to what you have - with the entries under the first "[INPUT]" block. You seem to have created multiple DCRs to track 3 different file paths?

Regarding configchunks, you are right - there should be a json file with details of log file path which is being attempted to be collected and additional details related to monitoring endpoints etc. It seems that the local config is not in sync with the DCR in portal.

In this scenario, here are some of the basic steps that can be taken to try download the latest config based on the configuration set in Azure DCRs. Note, not all the steps are mandatory to be followed - please try them in sequence and see if it helps with the issue.

1. Ensure that you have the latest Heartbeats from this machine to the tracked LA workspace. If not, the agents are not communicating. The query below can be used to check if latest heartbeats are available from the machine.

   Heartbeat
   | order by TimeGenerated desc
   

1. Check the "Extension" resource menu of this Arc server in Azure Portal. There should be an entry like below (to ensure that the extension is installed and healthy)

Ensure that you have created Data collection rules with the correct "data collection endpoints". Data Collection Endpoints facilitate the distribution of configuration from Azure to machine and also the log collection from machine to Azure. If they are not properly set, the issues may arise as we see in this case. For more details see point 3 & 4 in the following link - Create a data collection rule to collect data from a text or JSON file. Note, there are 2 locations where Data Collection Endpoints is specified when creating DCR. For details, see How to set up data collection endpoints based on your deployment

1. In case you have configured "Data collection endpoints" correctly with the DCR, and still don't see the logs getting collected, restart the AzureMonitorAgent to ensure that it downloads the latest config from the Azure platform. Please review the link that follows to get more details about the commands and meaning of configchunks - Linux AMA syslog agents: How to identify DCRs that are causing duplicate data collection

   cd /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-<agent version> 
   ./shim.sh -disable
   ./shim.sh -enable
   

1. After performing the step above, wait for a while and check the configchunks directory to see if json files corresponding to the DCRs were created. Also, the fluentbit.log file can be checked. Ideally, there should be entries like the following denoting the files being tracked for log collection in this log file after a restart.
2. In case, the steps above does not help, I would suggest deleting the DCR and recreating it again.
3. Finally, a reinstall of the Azure Monitor Agent, followed by recreating the DCR should help fix it.

In case, the steps mentioned above does not help, based on the nature of this issue, I would suggest reaching out to Azure Support to have this issue investigated by one of our support engineers, who can review the logs on the machine and platform logs if required to troubleshoot this further.

Please let me know if you have any questions. If the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.