Troubleshooting guidance for the Azure Monitor agent on Windows Arc-enabled server
Overview for Azure Monitor agent
Before you read further, you must be familiar with Azure Monitor agent and Data Collection Rules.
Terminology
Name | Acronym | Description |
---|---|---|
Azure Monitor Agent | AMA | The new Azure Monitor agent |
Data Collection Rules | DCR | Rules to configure collection of data by the agent, i.e. what to collect, where to send to, and more |
Azure Monitor Configuration Service | AMCS | Regional service hosted in Azure, which controls data collection for this agent and other parts of Azure Monitor. The agent calls into this service to fetch DCRs. |
Logs endpoint | -- | Endpoint for sending data to Log Analytics workspaces |
Metrics endpoint | -- | Endpoint for sending data to Azure Monitor Metrics databases. |
Instance Metadata Service and Hybrid | IMDS and HIMDS | Services hosted in Azure which provide information about currently running virtual machines, scale sets (via IMDS) and Arc-enabled servers (via HIMDS) respectively |
Log Analytics workspace | LAW | The destination in Azure Monitor that you can send logs collected by the agent to |
Custom Metrics | -- | The destination in Azure Monitor that you can send guest metrics collected by the agent to |
Basic troubleshooting steps (installation, agent not running, configuration issues)
Follow the steps below to troubleshoot the latest version of the Azure Monitor agent running on your Windows Arc-enabled server:
Carefully review the prerequisites here.
Verify that the extension was successfully installed and provisioned, which installs the agent binaries on your machine:
- Open Azure portal > select your Arc-enabled server > Open Settings : Extensions from the pane on the left > 'AzureMonitorWindowsAgent'should show up with Status: 'Succeeded'
- If not, check if the Arc agent (Connected Machine Agent) is able to connect to Azure and the extension service is running.
You should see the below output:azcmagent show
If instead you seeResource Name : <server name> [...] Dependent Service Status Agent Service (himds) : running GC Service (gcarcservice) : running Extension Service (extensionservice) : running
Agent Status: Disconnected
or any other status, file a ticket with Summary as 'Arc agent or extensions service not working' and Problem type as 'I need help with Azure Monitor Windows Agent'. - Wait for 10-15 minutes as extension maybe in transitioning status. If it still doesn't show up, uninstall and install the extension again and repeat the verification to see the extension show up.
- If not, check if you see any errors in extension logs located at
C:\ProgramData\GuestConfig\extension_logs\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent
on your machine - If none of the above works, file a ticket with Summary as 'AMA extension fails to install or provision' and Problem type as 'I need help with Azure Monitor Windows Agent'.
Verify that the agent is running:
- Check if the agent is emitting heartbeat logs to Log Analytics workspace using the query below. Skip if 'Custom Metrics' is the only destination in the DCR:
Heartbeat | where Category == "Azure Monitor Agent" and Computer == "<computer-name>" | take 10
- If not, open Task Manager and check if 'MonAgentCore.exe' process is running. If it is, wait for 5 minutes for heartbeat to show up.
- If not, check if you see any errors in core agent logs located at
C:\Resources\Directory\AMADataStore\Configuration
on your machine - If none of the above helps, file a ticket with Summary as 'AMA extension provisioned but not running' and Problem type as 'I need help with Azure Monitor Windows Agent'.
- Check if the agent is emitting heartbeat logs to Log Analytics workspace using the query below. Skip if 'Custom Metrics' is the only destination in the DCR:
Verify that the DCR exists and is associated with the Arc-enabled server:
- If using Log Analytics workspace as destination, verify that DCR exists in the same physical region as the Log Analytics workspace.
- On your Arc-enabled server, verify the existence of the file
C:\Resources\Directory\AMADataStore\mcs\mcsconfig.latest.xml
. If this file doesn't exist, the Arc-enabled server may not be associated with a DCR. - Open Azure portal > select your data collection rule > Open Configuration : Resources from the pane on the left > You should see the Arc-enabled server listed here
- If not listed, click 'Add' and select your Arc-enabled server from the resource picker. Repeat across all DCRs.
- If none of the above helps, file a ticket with Summary as 'DCR not found or associated' and Problem type as 'I need help configuring data collection from a VM'.
Verify that agent was able to download the associated DCR(s) from AMCS service:
- Check if you see the latest DCR downloaded at this location
C:\Resources\Directory\AMADataStore\mcs\configchunks
- If not, file a ticket with Summary as 'AMA unable to download DCR config' and Problem type as 'I need help with Azure Monitor Windows Agent'.
- Check if you see the latest DCR downloaded at this location
Issues collecting Performance counters
- Check that your DCR JSON contains a section for 'performanceCounters'. If not, fix your DCR. See how to create DCR or sample DCR.
- Check that the file
C:\Resources\Directory\AMADataStore\mcs\mcsconfig.lkg.xml
exists. If it doesn't exist, file a ticket with Summary as 'AMA didn't run long enough to mark and Problem type as 'I need help with Azure Monitor Windows Agent'. - Open the file and check if it contains
CounterSet
nodes as shown in the example below:
If there are no<CounterSet storeType="Local" duration="PT1M" eventName="c9302257006473204344_16355538690556228697" sampleRateInSeconds="15" format="Factored"> <Counter>\Processor(_Total)\% Processor Time</Counter> <Counter>\Memory\Committed Bytes</Counter> <Counter>\LogicalDisk(_Total)\Free Megabytes</Counter> <Counter>\PhysicalDisk(_Total)\Avg. Disk Queue Length</Counter> </CounterSet>
CounterSet
nodes, then the DCR wasn't parsed correctly. File a ticket with Summary as 'AMA unable to parse DCR config' and Problem type as 'I need help with Azure Monitor Windows Agent'.
Issues using 'Custom Metrics' as destination
Carefully review the prerequisites here.
Ensure that the associated DCR is correctly authored to collect performance counters and send them to Azure Monitor metrics. You should see this section in your DCR:
"destinations": { "azureMonitorMetrics": { "name":"myAmMetricsDest" } }
Run PowerShell command:
Get-WmiObject Win32_Process -Filter "name = 'MetricsExtension.Native.exe'" | select Name,ExecutablePath,CommandLine | Format-List
Verify that the CommandLine parameter in the output contains the argument "-TokenSource MSI"
Verify
C:\Resources\Directory\AMADataStore\mcs\AuthToken-MSI.json
file is present.Verify
C:\Resources\Directory\AMADataStore\mcs\CUSTOMMETRIC_<subscription>_<region>_MonitoringAccount_Configuration.json
file is present.Collect logs by running the command
C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\<version-number>\Monitoring\Agent\table2csv.exe C:\Resources\Directory\AMADataStore\Tables\MaMetricsExtensionEtw.tsf
- The command will generate the file 'MaMetricsExtensionEtw.csv'
- Open it and look for any Level 2 errors and try to fix them.
If none of the above helps, file a ticket with Summary as 'AMA unable to collect custom metrics' and Problem type as 'I need help with Azure Monitor Windows Agent'.
Issues collecting Windows event logs
- Check that your DCR JSON contains a section for 'windowsEventLogs'. If not, fix your DCR. See how to create DCR or sample DCR.
- Check that the file
C:\Resources\Directory\AMADataStore\mcs\mcsconfig.lkg.xml
exists. If it doesn't exist, file a ticket with Summary as 'AMA didn't run long enough to mark and Problem type as 'I need help with Azure Monitor Windows Agent'. - Open the file and check if it contains
Subscription
nodes as shown in the example below:
If there are no<Subscription eventName="c9302257006473204344_14882095577508259570" query="System!*[System[(Level = 1 or Level = 2 or Level = 3)]]"> <Column name="ProviderGuid" type="mt:wstr" defaultAssignment="00000000-0000-0000-0000-000000000000"> <Value>/Event/System/Provider/@Guid</Value> </Column> ... </Column> </Subscription>
Subscription
nodes, then the DCR wasn't parsed correctly. File a ticket with Summary as 'AMA unable to parse DCR config' and Problem type as 'I need help with Azure Monitor Windows Agent'.
File a ticket
- Open a data collection rule and on the menu on the left, select New Support Request. You can also open the Help + support pane and select Create a support request.
- Select:
- Issue type: Technical.
- Subscription: Select the subscription where your machines reside.
- Service type: Data collection rules and Azure Monitor Agent.
- Is your issue related to a resource?: Yes. Select your machine by using the resource picker.
- Enter Summary and Problem type as indicated from the troubleshooting steps. Accurate information leads to faster resolution of the issue.
- Select Next and review the recommended solutions to see if they help.
- If they don't help, select Next and fill in the next set of details.
- Select Next, review the final details, and select Create.
Pripomienky
https://aka.ms/ContentUserFeedback.
Pripravujeme: V priebehu roka 2024 postupne zrušíme službu Problémy v službe GitHub ako mechanizmus pripomienok týkajúcich sa obsahu a nahradíme ju novým systémom pripomienok. Ďalšie informácie nájdete na stránke:Odoslať a zobraziť pripomienky pre