In this scenario we had some Issues with VM replication on Azure. So customer had removed the Protection completely and he was trying to configure it again. In this process he had also uninstalled the ASR Provider as well from the Hyper-V server and even the ASR Agent was also uninstalled it seems.

So we installed the Provider first on the Hyper-V host and then we were trying to install the MARS Agent, but that was just failing to even launch the Wizard.

After researching a bit on this issue, we found out that this type of issue can occur if the following registry key exists on the Hyper-V Host while installing the MARS Agent:

HKCR\Installer\UpgradeCodes\9E3CC81A48FC05646B4407520C8CC01D

We searched for the above Registry key and it was present on the Server. SO we went ahead and Renamed the key 9E3CC81A48FC05646B4407520C8CC01D To 9E3CC81A48FC05646B4407520C8CC01D_OLD and then we tried to re-install the MARS Agent again on the Hyper-V server. This time we were able to launch the Agent installation Wizard successfully and we could configure it fine as well.

So, this was one more case where it proved that how the Stale Registry entries can cause issues with the new installations or set ups.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

The set up in this case was that we had a two node Windows Storage Server 2012 R2 based Scale Out File Server Cluster.

When we were trying to add this SOFS Storage in VMM using the "Add Storage Devices Wizard" we were getting the following error at the stage "Discover and import storage device information":

Storage discovery of provider {0869e333-12f6-44c7-b6b7-7139530a18c9} for user usramdom\svc.vmm failed from 101VMM01.magnetek.com with error code 46010.

Specify valid provider, port and user credentials for storage discovery. ID: 26101.

As we do on most of the VMM cases , we collected the VMM Debug ETL tracing in this case as well using the steps mentioned in the article below:

https://support.microsoft.com/en-us/kb/2913445 

After collecting the VMM logs, we analysed the trace logs and saw the errors below:

569 [0]053C.139C::‎2016‎-‎01‎-‎28 11:04:23.065 [Microsoft-VirtualMachineManager-Debug]4,4,SCXStorageService.cs,1578,Found provider TargetServer = , Name = Storage Spaces Management Provider, Type = 1, ObjectId = {1}\\101VMM01\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageProvider.ObjectId="{0869e333-12f6-44c7-b6b7-7139530a18c9}", URI = , UniqueId = {0869e333-12f6-44c7-b6b7-7139530a18c9}:101VMM01,{00000000-0000-0000-0000-000000000000}

580 [0]053C.139C::‎2016‎-‎01‎-‎28 11:04:24.174 [Microsoft-VirtualMachineManager-Debug]4,4,WsmanAPIWrapper.cs,2436,WinRM: URL: [http://101vmm01.magnetek.com:5985], Verb: [INVOKE], Method: [Discover], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/microsoft/windows/storage/MSFT_StorageProvider?ObjectId={1}\\101VMM01\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageProvider.ObjectId="{0869e333-12f6-44c7-b6b7-7139530a18c9}"],{00000000-0000-0000-0000-000000000000}

1051 [0]053C.139C::‎2016‎-‎01‎-‎28 11:04:27.252 [Microsoft-VirtualMachineManager-Debug]4,4,SCXStorageService.cs,1984,SCX SMIS operation on provider {1}\\101VMM01\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageProvider.ObjectId="{0869e333-12f6-44c7-b6b7-7139530a18c9}" failed with   CIMStatusCode    CIMStatusCodeDescription  from   ErrorSource  of   ErrorSourceFormat  and   ErrorType  with   Message    MessageArguments    MessageID    OtherErrorSourceFormat    OtherErrorType    PerceivedSeverity    ProbableCause    ProbableCauseDescription    RecommendedActions    OwningEntity ,{00000000-0000-0000-0000-000000000000}

Then I did some more research on the above Errors and found out that this type of issue can occur if there was an SOFS storage added earlier to the VMM server and was later removed, but it leaves some Registry entries behind, which causes this issue.

We had to look for ConnectionSubsystems Registry keys on the VMM server under the location below:

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\StorageManagement\SpacesSMP\ConnectedSubsystems

And there we found out that we had some entries from your Old SOFS cluster which was earlier added to VMM but then later removed from there. But these registry entries were still hanging in there due to some reason.We took the complete Backup of the Registry on VMM. Then we deleted the stale entries of the Old SOFS Cluster from the above Registry Key and then we rebooted the VMM server.

After that when we tried to add the new SOFS Storage on the VMM server, we could add it successfully now.

So, as you can see, the main issue here was the stale Storage registry entries which were still sticking there in the Registry due to some reason.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

This Post will take you through an issue where a VM Protected using ASR in a Hyper-V to Azure scenario, keeps on going in a Resynchronization state randomly. And when we try to ReSynch, it fails

We see the below Error for the VM on the Hyper-V Host On-Prem:

And then the ReSynch was also failing with the Error below:

While reviewing the configuration, we came to know that this VM had two Disks, the OS Disk was of around 100 Gigs and the Data Disk was Exactly 1 TB in Size.

According to the Article below the Maximum Disk Size supported for the Data Disk in ASR is 1 TB:

http://azure.microsoft.com/en-us/blog/protecting-large-data-disks-using-azure-site-recovery/ 

So, looking at the above limitation of ASR, we decided to Shrink the Data Disk Size by around 100 GB, so that it can come down to 900 GB. And as soon as we did that, we did not see the VM going into the Resynchronization state anymore.

We monitored it for a few days and we still didn’t see any issues.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

Here is another very interesting issue that I worked on a few days back.

Let me first give you some history about the environment. We had a Few Untrusted Hosts in the perimeter network added to VMM and we tried to update the Host Agent to UR 6 on those Hosts, as the VMM was upgraded to UR 6 as well. But when we tried to upgrade the Agent of those Hosts it gave the below error:

Error 10436: Virtual Machine Manager does not support updating or re-associating an agent on a host that is in a non-trusted domain or on a perimeter network.

Which was expected according to the Blog Post below:

http://blogs.technet.com/b/scvmm/archive/2015/05/19/how-to-update-untrusted-hosts-in-system-center-2012-r2-virtual-machine-manager.aspx

But then the customer removed those Hosts from Cluster from VMM. And then he was trying to add them back. But now the Host machines were not even getting added back to the Cluster and they stayed in Pending status. Also, we exported the Host Certificates from the Hosts to VMM’s Trusted People Store as it was not happening automatically. And we also updated the VMM agent on the Hosts to UR 6 version manually as well.

And then we again tried to Add the Hosts back to the Cluster in VMM, but it again failed. So, we collected the VMM traces while trying to add the Hosts in the cluster and we saw the below error there:

SecureCredential savedCredential, String& certificate) *** Carmine error was: CannotRetrieveCertificateError (20909); HR: 0x80072746 *** Host1.Contoso.Com ** The underlying connection was closed: An unexpected error occurred

And the Error Code 0x80072746 translates as below:

err 0x80072746

# as an HRESULT: Severity: FAILURE (1), FACILITY_WIN32 (0x7), Code 0x2746

# for hex 0x2746 / decimal 10054

  WSAECONNRESET                                                  winerror.h

# An existing connection was forcibly closed by the remote

# host.

# 1 matches found for "0x80072746"

As we can see above error is showing that the Remote Host is closing the connection. So to check further on why the Host is closing the connection, we collected a simultaneous Network Capture on the VMM server and the Host while trying to reproduce the issue. And here is what we saw in the Network Capture:

Network Capture on VMM:

Same Communication on the Host:

So as we can see in the above Network Captures, we are getting a SSL Client HELLO on the host from the VMM server, but the Host is not responding with a SERVER HELLO on top of that, which we should expect to see after that. And that’s what looks to be the root cause here.

And because we see the behaviour mentioned above, we had to collect the WinRM listener’s output from the Host and as expected it shows that it is NOT listening on HTTPS on Port 5986, it only had a listener for HTTP Port 5986 as shown below and that is why it doesn’t have the Certificate Bound to it as well:

Output for the command  winrm e winrm/config/listener:

Listener

    Address = *

Transport = HTTP

Port = 5986

    Hostname

    Enabled = true

    URLPrefix = wsman

CertificateThumbprint  <<<< It is Blank

    ListeningOn = 127.0.0.1, 169.254.2.213, Other IP Addresses

We should have expected to see a WinRM Listener for HTTPS Protocol on Port 5986, bound to a valid certificate, which was not the case here.

So we followed the following action plan to delete the current listener and create a new WinRM listener for Port 5986, for the correct protocol HTTPS  and bound to a valid certificate of the Host:

- WinRM Listener configuration was adjusted

o Deleted the listener

winrm delete winrm/config/listener

o Recreated the listener

winrm create winrm/config/Listener?Address=*+Transport=HTTPS  @{Hostname="<serverfqdn<";CertificateThumbprint="<SCVMM Agent certificate thumbprint>"}

After doing this we could see the listener correctly on the Host. Then we re tried to add the Hosts back to the VMM server and it was successful this time.

It was a lot of work done on this Issue, that’s why I wanted to share it with you all, so that if some one else hits the same issue, he would not have to put all these efforts again in resolving it.

I hope it will be helpful.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

So I am back to share some more interesting issues with you all.

In this particular case we were trying to add a Host machine to SCVMM 2012 R2. And although the Server was getting added in VMM, we were seeing the Error below in the Jobs:

Error (21201)

Another machine with the same SMBIOSGUID is found.

Recommended Action

An SMBIOS GUID should uniquely identify the machine. Please provide the correct value or contact the machine manufacturer to help update the hardware with correct information.”

So we removed the Host from VMM and we gathered the VMM traces while trying to add the Host back. And here is what we saw in the Logs:

649597 [1]1384.199C::‎2015‎-‎06‎-‎09 16:43:13.243 [Microsoft-VirtualMachineManager-Debug]4,4,SqlContext.cs,727,Disposing (SqlContext#264d9) {[rw] AddHostSubtask.RunSubtasktx = (SqlTransaction#7af68b) System.Data.SqlClient.SqlTransaction, currentTxRecord = (SqlTransactionRecord#22e) commandList: { n=1 { tx:(SqlTransactionRecord#22e) NQ:"dbo.prc_PMM_PhysicalMachine_Update(@Id=52f61feb-cf0e-4b91-b44f-cfb70683590c, @BMCType=0, @BMCModule=, @BMCAddress=, @BMCPort=, @SMBiosGuid=xxxxxxxxxxxxxxxxxx)", @6/9/2015 4:43:13 PM [+00:00:00.0468823] } } txRecordList: { n=1  { tx#558 AddHostSubtask.RunSubtask() +00:00:00.1103098 ReadCommitted canRetry: false commands: { n=1 { tx:(SqlTransactionRecord#22e) NQ:"dbo.prc_PMM_PhysicalMachine_Update(@Id=52f61feb-cf0e-4b91-b44f-cfb70683590c, @BMCType=0, @BMCModule=, @BMCAddress=, @BMCPort=, @SMBiosGuid=375d451b-9a1f-4b15-a4f0-9c70b1a7566b)", @6/9/2015 4:43:13 PM [+00:00:00.0468823] } } } } , conn = (SqlConnection#1a31af) { State: Open Database: VirtualManagerDB DataSource: Server.contoso.com\SCVMM,1433 PacketSize: 8000 ServerVersion: 11.00.3153 StatisticsEnabled: False WorkstationId: USATRAME0914 } },{00000000-0000-0000-0000-000000000000}
649630 [2]1384.199C::‎2015‎-‎06‎-‎09 16:43:13.328 [Microsoft-VirtualMachineManager-Debug]4,2,Task.cs,287,CarmineException [ex#36e] caught by SubtaskRun (catch CarmineException) [[(CarmineException#3368) { Microsoft.VirtualManager.Utils.CarmineException: Another machine with the same SMBIOSGUID is found.  An SMBIOS GUID should uniquely identify the machine. Please provide the correct value or contact the machine manufacturer to help update the hardware with correct information. --->

Looks like VMM Is trying to Update the Stored Procedure "dbo.prc_PMM_PhysicalMachine_Update" in SQL Database with the Object ID 52f61feb-cf0e-4b91-b44f-cfb70683590c and the SMBIOSGUID and that's where it fails.

So we followed the action plan below to first find the UUID of the Cluster node and then removed the existing entry of that Host from the DB:

1. Open an elevated command prompt and run wmic csproduct get uuid on the Cluster node with which we have the issue

2. That will give us the SMBIOS ID for that server

3. Check that the UUID exists in the SCVMM DB under the tbl_PPM_Physicalmachine  table.

4. Once you verify that the above UUID exists, take a VMM DB backup and then delete the UUIDs from the VMM db.

And after following the action plan, we were able to add the Host back to VMM, without any issues and Errors.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

Friends, I am back with yet another Post on SCVMM(System Centre Virtual Machine Manager). It is very simple but still important as it wont tell you about the real issue anywhere in the Errors which you will see on the screen.

So, the issue starts from inability to Launch the VMM Console and we see Errors like below:

If we go by the above Errors we will think that it’s a Permissions issue and the user account has gone missing from the RunAs Accounts in VMM.

When you go and check the Local Administrators Group on VMM server you see something like this where instead of the User Accounts we see GUIDs, which might tell that there is some issue going on with the Domain Connectivity:

From here we got a clue and we ran the below command to check the Secure Channel to the Domain Controller and that also failed as expected:

nltest /sc_verify: Domain.com

Then we checked the Event Viewer and we saw the below Events coming there:

After looking at the above Errors, it was confirmed that the issue was with the Domain Connectivity. So we worked on that and after fixing the domain connectivity issue, we were able to Open the VMM console fine.

I hope this will help you addressing similar issues.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

This is another interesting case on which I worked recently where we needed to scratch the surface in VMM to know what the Root Cause was.

We were trying to migrate a VMs in SCVMM from one Host to the other Host. And as both the Hosts had SAN storage configured, most of the VMs were showing the correct “Transfer  Type”. But while migrating a few other VMs we could see the “Transfer Type” as “Network” instead of “SAN”:

And we were also getting the Message for “Deployment and Transfer Explanation” as shown above and it said “Virtual Machine resides on a LUN which is not visible to any of the Storage Providers. Which was actually very strange because all the settings for storage were configured correctly.

So to understand what exactly is happening here, we collected the VMM tracing while trying to reproduce the issue as per the Article below:

https://support.microsoft.com/en-us/kb/2913445/en-us 

And after we analysed the VMM trace we saw the below message appearing there under the Function SANMigrationBetweenHAandNonHAHostPrecheck.RunPrecheck:

DeployLUNNotFoundUsingStorageProvider (26207)

And also we saw the Below Paths being referenced there:

[Microsoft-VirtualMachineManager-Debug]4,4,SANStorageManagementHelper.cs,1342,filePath C:\Windows\system32\vmguest.iso accessPath \\?\Volume{21fd0518-ea71-11e4-80cb-bc305bee5327}\

So as you can see in the above path, it is trying to access VMGuest.ISO somehow for this VM which is Ofcourse not SAN Capable or on a SAN storage. SO went back and checked the VM settings in VMM to see if that ISO is Mounted on that VM. But to our surprise that VM did not have any ISO mounted.

Our further investigation revealed that the VM had a few Check Points configured and those Check Points had this VMGuest.ISO mounted there as shown below:

So we deleted those CheckPoints from the VM configuration in VMM and after that when we tried to migrate, we now saw the “SAN” option showing up there as “Transfer Type”.

As you can see that even if at the surface everything looks fine you can still have some issues in the configuration deep inside causing Problems like this.

Will catch you later in my upcoming Posts…Happy Reading.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

This was a very interesting and weird issue which I worked upon a few days back.

We had a few Hosts running 2008 R2 VMs and 2012 R2 VMs. We had a mix of Virtual Networking as well. Some of the VMs were using the VLAN enabled Networks. And some other were using NVGRE Networking, where the VMs Network will point to another VM which acts as a Gateway to the Internet to these VMs. You can get more information about NVGRE in SCVMM in the article below:

http://blogs.technet.com/b/scvmm/archive/2013/09/17/whitepaper-hybrid-clouds-with-nvgre.aspx 

So the issue was that all the Routing was working fine from all the VMs. But only part which was not working was the Name resolution and too only from 2012 R2 VMs. The name resolution was working fine from 2008 R2 VMs. Which was very odd, as if it would have been an issue with NVGRE configuration in VMM, then even normal Routing would not have worked. But we still went ahead and checked the complete configuration for NVGRE and everything looked to be properly setup.

Then we collected some Network Captures on the Gateway VM while trying name resolution from one of the 2012 R2 VM, and we could see the DNS Queries coming there and in fact we could see those DNS Queries even going out to the Public DNS servers, as shown below:

As we can see above the DNS Queries were going out but there was no response which was coming back or atleast we can not see any response in the Network captures.

So we did some research on this weird behavior and we found the below Article from one of our MVPs Thomas Maurer:

http://www.thomasmaurer.ch/2014/03/hyper-v-network-virtualization-nvgre-no-connection-between-vms-on-different-hyper-v-hosts/ 

And we followed the action plan suggested by Thomas in the above Article to Disable the “Encapsulated Task Offload” on the Broadcom NICs on these Hosts where the 2012 R2 VMs were running.

After that the name resolution started working fine on those 2012 R2 VMs as well.

So, as you saw, it was one of those cases where NIC drivers can cause a lot of weirdness.

Again thanks to Thomas Maurer for his Excellent Article which helped us in resolving this issue.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division

Microsoft Security and Manageability Division

Here is another interesting VMM issue on which I worked recently.

We were getting the following Error in the VMM Jobs while trying to deploy a VM:

Error (2912): "An internal error has occurred trying to contact the <HOST_NAME> server: NO_PARAM: NO_PARAM

This Error can come if the VMM certificate of the managed Hyper-V host in the VMM server certificate store does not match with the VMM certificate on the managed host. 

So we collected the VMM Tracing to see what’s happening here and here is what we saw in the Trace:

02.c01.local:5985], Verb: [INVOKE], Method: [AddPeerCertificate], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement],{00000000-0000-0000-0000-000000000000}

[2]06E8.15A0::?2015?-?03?-?23 18:26:28.161 [Microsoft-VirtualMachineManager-Debug]0,4,CertificateServices.cpp,1104,CCertificate::GetSubjectName: certificate issued to: [Host1.contoso.local],{00000000-0000-0000-0000-000000000000}

[2]06E8.15A0::?2015?-?03?-?23 18:26:28.161 [Microsoft-VirtualMachineManager-Debug]0,4,CertificateServices.cpp,165,CCertificate::GetRightRootStoreName: store name according to platform: [TRUSTEDPEOPLE],{00000000-0000-0000-0000-000000000000}

[2]06E8.15A0::?2015?-?03?-?23 18:26:28.161 [Microsoft-VirtualMachineManager-Debug]0,4,CertificateServices.cpp,1153,CCertificateServices::GetSCVMMCertificate: looking for certificate for [Host1.contoso.local] in [TRUSTEDPEOPLE],{00000000-0000-0000-0000-000000000000}

As we can see that VMM is trying to look for the Certificate for the Managed Host in its TRUSTEDPEOPLE Store.

We can check the Certificate on the VMM server by navigating to the certificate store (computer account) and check the certificate of the managed host under the Trusted People -> Certificates. Check the Thumbprint on that Certificate and then go to that Host and check the Certificate under Personal -> Certificates, if the Thumbprint of this Certificate and the Thumbprint on this Host’s certificate on the VMM server matches, then its fine, but if it doesn’t match, then it can cause this kind of an issue.

I have shown an example of how the Certificates should like in an ideal scenario.

This Cert below is on VMM server’s “TrustedPeople” Store of one of the Managed Hosts.

And the Cert below is on the Managed Host itself in its Computer’s Personal Store.

So, as you can see the Thumbprint of both the Certificate matches. And that’s how it should be. If the Thumbprint of these two Certs doesn’t match then we can get into an issue which we have discussed here.

However in this case the Certificates did not match as expected.

To fix the issue we had to run the below PowerShell commands:

$credential = get-credential

Get-VMMManagedComputer -ComputerName "HostName" | Register-SCVMMManagedComputer -Credential $credential

NOTE: After the first command above we will have to provide the Local Administrator’s Credentials of the Host when prompted.

And after the second command VMM will Import the Host’s Certificate from the Host to the VMM server’s TRUSTEDPEOPLE store.

After performing the above steps, when we tried to deploy the VM, it got Deployed fine.

Author:

Nitin Singh

Support Escalation Engineer

Microsoft Security and Manageability Division