Troubleshooting the On-Demand Assessments (MMA)

  • Article

Download the Assessment Troubleshooting Script

Please download and run the Assessment Troubleshooting script to self-evaluate and troubleshoot the On-Demand Assessments. Refer to the full troubleshooting guide below:

The below steps would walk you through from start to end and make you verify the correctness of each requirements that are to be met in running the On-Demand Assessments:

The most common issues we see users encounter are: (Guidance on how to resolve them is mentioned below in the article)

  1. When you run the assessment but see no data in Log Analytics -> Restart healthservice if data files are pending ingestion.

  2. Error message: "You don't have access to Azure Log Analytics" in Services Hub -> IT Health -> On-Demand Assessments.

Microsoft Monitoring Agent (MMA) Installation issues

Sometimes when installing MMA Agent, it cannot successfully connect to the workspace when specifying the credentials (Id and Key) of the Azure Log Analytics (OMS) workspace. A workaround which sometimes works is to remove the WorkspaceId and the key and install MMA without specifying any workspace. Then, open the MMA control panel by going to Control Panel, Security & Settings, Microsoft Monitoring Agent, Azure Log Analytics (OMS) tab and add the desired workspace by specifying its ID and Key. Then click the Apply button and verify that the green checkmark appears.

Linking and Permissions

  1. Click this link to watch the video to pre-configure your On-Demand Assessments.

  2. Verify that you have the Azure Subscription Owner role on the Azure Subscription on the same email ID that you use to login into Services Hub.

  3. You should be able to see the below page in Services Hub under the Health tab then Assessments upon successful linking.

  4. Confirm that the Log Analytics workspace you have access to is the one that is linked in Services Hub. If not, ask them to relink by clicking on profile at the top right then click Edit Log Analytics Workspace and link the desired workspace.

  5. Confirm that you have added the desired assessment from the catalog.

    Assessments results dashboard with different tabs for Skype, SharePoint, System Centers Management and Entra ID.

    Available On-Demand Assessments with listing of available items such as SharePoint, Entra ID, Exchange Server, etc.

    Active Directory Assessment description for supporting Entra ID service Environments.

Some of the most frequent problems when a command such as AddExchangeAssessmentTask is invoked, are described next:

Windows Server 2008 R2 does not recognize Add-*AssessmentTask commandlets

It is recommended not to run assessments on a Windows Server 2008 machine. But when you absolutely need to, you need to do the following:

  1. Install latest version of Powershell and .net 4.6.2 on the Windows 2008 server. Most Windows 2008 Servers have Powershell which uses .net 2.0 and that does not meet the requirements.
  2. Run Powershell in Administrator mode.
  3. Before invoking any Add-*AssessmentTask commandlets issue: Import-Module $env:ProgramFiles\'Microsoft Monitoring Agent\Agent\PowerShell\Microsoft.PowerShell.Oms.Assessments\Microsoft.PowerShell.Oms.Assessments.dll'

After this command is executed, you should be able to invoke Add-*AssessmentTask commandlets.

On any platform, if the Add-*AssessmentTask commandlets are not recognized

  1. Verify that the OMS Assessments Powershell module has been downloaded. To do this, verify that $env:ProgramFiles\'Microsoft Monitoring Agent\Agent\PowerShell' has a subdirectory named Microsoft.PowerShell.Oms.Assessments and in that subdirectory there is a file named Microsoft.PowerShell.Oms.Assessments.dll, like shown in the screenshot below:

    PowerShell Window demonstrating correct Working Directory and Subdirectory.

  2. If Microsoft.PowerShell.Oms.Assessments.dll is not there, then make sure that your Log Analytics workspace is linked from Services Hub.

  3. Verify that $env:PsModulePath contains $env:ProgramFiles\'Microsoft Monitoring Agent\Agent\PowerShell'.

  4. Verify that the installed version of PowerShell is at least 4.0 (Just type $PsVersionTable in the Powershell window) and that Powershell uses CLRVersion equal to or greater than 4.0.

Troubleshooting Assessment Installation Errors when executing an Add-*AssessmentTask cmdlet

  1. Read the log file.

  2. The log file location is displayed on the Powershell console window. It is just an informational message -- in white. See this location in the sample screenshot below:

    PowerShell Windows displaying file location.

Note

If Add-_AssessmentTask fails for some reason, look at the log file.

  1. The -ScheduledTaskUserName and -ScheduledTaskPassword may be invalid -- no such user exists or the password is invalid, or has expired.

    PowerShell Windows displaying user error message.

Requirements for successfully running the scheduled task

Verify the user account Group Policies: Logon as Batch Job Permission

Note

At times, the assessment may not get triggered from the Task Scheduler. This may happen if the user does not have running batch job permission. If that’s the case, this permission needs to be explicitly granted by going in here from gpedit.msc.*

Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment

  1. Right click on "Log on as batch job" and select Properties.
  2. Click "Add User or Group" and include the relevant user.

Do not forcefully unload the user registry at user logoff

On the data collection machine, change the following setting in the group policy editor (gpedit.msc) from "not configured" to "enabled"; go to Computer Configuration, Administrative Templates, System, and then User Profiles.

'Do not forcefully unload the user registry at user logoff'

Disable the FIPS Policy

  1. In Control Panel, click Administrative Tools and then double-click Local Security Policy.

  2. In Security Settings, expand Local Policies and then click Security Options.

    Local Security Policy window displaying Security Options folder.

  3. Under Policy in the right pane, double-click System cryptography; use FIPS compliant algorithms for encryption, hashing, signing, and then click Disabled.

Network Access: Do not allow storage of passwords and credentials

  1. This error occurs with the message, "A specified logon session does not exist. It may already have been terminated."

  2. To resolve this, go to SECPOL.MSC, Security Settings, Local Policies, then Security Options.

  3. Do not allow storage of passwords and credentials for network authentication.

    Local Group Policy Editory window.

  4. Set the policy to disabled.

Assessment has not been added to your workspace

If invoking the start-XXXAssessment results in this error: "There is no file XXXAssessment.execPkg associated to the current Log Analytics workspace ... " as in the screenshot below, the reason is that the specific assessment-type must be added to the Log Analytics workspace to which the Microsoft Monitoring Agent is currently connected. Add the assessment on the ServicesHub page. (Dropdown the Health tab and then choose Assessments. At the end of the page, click on the desired assessment that is listed under "Available On-Demand Assessments" and then on the modal dialog that appears for adding the selected assessment, click "Add Assessment") Then wait at least 10 minutes before attempting to start the assessment. The delay is required to allow the MMA Agent to contact Log Analytics and download the Management Pack which contains the XXXAssessment.execPkg.

PowerShell window with a no file associated error message.

Assessment Task Running Issues

Inactive / No Data found in Azure Log Analytics

Verify Log Analytics Agent connectivity

To ensure that the agent can communicate with Azure Log Analytics, go to Control Panel, Security & Settings, and Microsoft Monitoring Agent. Under the Azure Log Analytics (OMS) tab, look for a green check mark.

Microsoft Monitoring Agent Properties window displaying green check mark.

A green check mark icon confirms that the agent is able to communicate with the Azure service.

A yellow warning icon means the agent is having issues communication with Log Analytics.

Look at the Heartbeat messages from the AgentHealthAssessment solution

When you add the AgentHealthAssessment solution to your Log Analytics workspace through Azure Portal, you will see useful information, including a list of computers with unresponsive agents and the last successful-heartbeat time for each of these unresponsive machines.

Log Analytics dashboard displaying heartbeat messages.

If your machine is listed there, one common reason for this is that the Microsoft Monitoring Agent service (HealthService) has stopped. Use service control manager to restart the service.

If you have a firewall restriction in place, make sure the below ports are opened up:

This can easily be verified by invoking the TestCloudConnection.exe tool which is in the Microsoft Monitoring Agent\Agent\ folder, as in this screenshot:

Command Prompt for VS2015.

Data from OnDemand assessment is no longer seen in Log Analytics, but it was seen in the past

Find the scheduled task in the Task Scheduler and run the task manually from the Task Scheduler. Verify that an OmsAssessment.exe process is running. If no such process is running then one possible reason is that the password specified at the time the Add-XXXAssessmentTask was run for this assessment is no longer valid (e.g. has expired or has been changed) If this is the case, you will be able to see the error both on the History tab of the Task Scheduler, and also in the Task Scheduler Event Log -- with EventId 101 (and 104) and ErrorValue: 2147943726 -- as shown on the next two screenshots:

Task Scheduler Launch Failure.

Task Scheduler Launch Failure Error Details.

Restart healthservice if data files are pending ingestion

Please close all active PowerShell windows on the machine. Now, if you check the working directory of the Assessment and find the files with names like new.recommendations.*** (see screenshot below):

Open Command Prompt in Administrator mode and type in:

net stop healthservice net start healthservice

After running the below command, the files would change to be processed as shown below which means the files have been ingested successfully and data should be visible on Log Analytics in about 30 minutes.

Assessments folder with numbered folder highlighted.

Command Prompt dialog.

Assessments folder and targets.

Check for any conflicting omsassessment.exe processes running

Open up Task Manager and look for a process named as omsassessment.exe. If found it indicates that the assessment is still running.

If it has been quite long (for eg, if you find this process has been running for over a day), it is possible that the assessment agent could not process data. So please proceed to the next troubleshooting steps below.

Go through any errors in the prerequisite file

Go to the assessment working directory and look at the pre-requisites (processed.prerequisites) files to find any errors mentioned for the assessment targets.

Assessments folder process.prerequisites.

If any errors are found, for example WMI connectivity issues, the target names and the error would be mentioned in this file. Please resolve this and then trigger the assessment; from the Task Scheduler, expand Microsoft, Operations Management Suite, AOI***** , and then Assessments to right click on the desired assessment scheduled task and click run.

Go through error in the discovery log file

Go to the assessment working directory and go into the 6-8 digit numbered folder inside the directory. Look for a folder called as Logs within which you will find a file named as DiscoveryTrace*** .

Look for any errors or exceptions in this file and resolve them since they would be related to credential/permissions issue, WMI failure, network issue etc.

Large file ingestion

If the below files processed.recommendations.*** size is greater than 250MB, the files might be difficult to be processed by the Log Analytics Agent. If you encounter this scenario and are not able to see the data in Log Analytics please contact serviceshubteam@ppas.uservoice.com and let us know about your issue.

Assessments folder displaying file size.

Try to reduce the number of targets per assessment schedule

If you are running the Windows Server, Windows Client or SQL Assessment and have added more than 5 targets in a single scheduled task, sometimes its possible that the assessment agent would not be able to process so many targets in one go. If you encounter this scenario, please use the below cmdlet to remove any existing configuration:

Remove-WindowsClientAssessmentTask Remove-WindowsServerAssessmentTask Remove-SQLAssessmentTask

Now run the Add-AssessmentTasks again with fewer targets. You can add multiple such tasks and create batches of tasks with 3-5 targets per task which would result in a quicker evaluation of your entire environment.

Go through Scheduled Task dispatch and uploader log files

In the Assessments working directory, there is a \Logs\ folder which contains *Commandlet*.log and *Module*.log. The *Commandlet*.log file contains data about the scheduled task starting the Powershell commandlet and this can be used to find why the scheduled task did not start. If this file is not produced when the ODA scheduled task is started, it usually indicates a password error.

The *Module*.log file contains data about the Health Service and its attempts to upload data to the Azure Log Analytics cloud.

The Applications and Services Logs\Operations Manager Event Log also contains information which can be used to troubleshoot various issues.