So here's my symptoms:

Problem: When adding a new subscription to a new user the status of that subscription appears Active (Out of Sync).

Noted Event Errors:

Event ID 262 - MgmtSVC-AdminAPI/Operational (Example: "Job 'fdce0927-9137-4346-8616-999aa8a8b962' failed: System.Exception: Tenant for subscription cf02b3f9-1598-41bb-8b61-5f4f039221c0 does not exist.")
Event ID 204 - MgmtSVC-AdminAPI/Operational (Example: "Resource provider unexpected exception for fanout request with verb 'POST', operation name 'Fanout', request URI 'https://localhost:8090/provider/subscriptions',  version '', client request Id '', server request Id '', status code 'InternalServerError', reason 'Internal Server Error', body 'Failed to create subscription. Reason: Message : An error occurred while processing this request., InnerMessage: <?xml version="1.0" encoding="utf-8"?>")

Another symptom of this issue is, when you create and assign the new user to the subscription the account is not created in VMM, under Settings/Security/User Roles.

Working through the symptoms I narrowed it down to being an issue with SPF (Service Provider Foundation) communicating to VMM. I checked the localspf user permissions and all appears fine.

Looking in IIS, I checked over the SPF site. The Binding appeared fine...but when looking at the account used to run the Provider, it appeared as Network Service (obviously over-looked during setup!). Changing this "Identity" to a domain account and after hitting "Recycle" on the Provider I was able to successfully create a new user and assign a subscription. Which resulted in the new account appearing in VMM.

IIS Manager -> Application Pools:

Useful Azure Pack setup blog: https://blogs.technet.com/b/privatecloud/archive/2013/12/06/windows-azure-pack-installing-amp-configuring-series.aspx

Check_installed_IS.zip

Having finished rolling around laughing! I began to look into the calculations behind the feature. Trying to find good information on this feature, at the time, was extremely difficult.

In an attempt to publicize this feature further I would simply like to reiterate the information now available in a blog article by Hilton Lange: https://blogs.technet.com/b/scvmm/archive/2014/12/15/taking-a-closer-look-at-the-virtual-machine-manager-cluster-overcommit-algorithm.aspx

In Hilton's blog he talks about the 2 methods (Simple & Proof) and the 4 checks (Proof-simple, Slot-simple, Proof-full & Slot-full), VMM undertakes to attempt to prove that a cluster is not over-committed. If any 1 of the checks results in an "OK" (cluster not over-committed) VMM will mark the cluster as "OK".

So all of this information is great, but how do you know WHY or WHEN your cluster is likely to be Over-committed? The answer to both these questions would normally involve a lot of numbers and manual calculations. However following on from a PowerShell script I discovered (https://www.miru.ch/unveiling-cluster-overcommit-in-scvmm-2012-hyper-v/) I have created a method of checking all clusters, within a single HostGroup, to confirm if they are or indeed when they are likely to become over-committed. The PowerShell script only performs the Proof-full check on HostsGroups with a Cluster Reserve set to 1. Using the output report you will have the data to manually check the remaining methods and indeed in some cases where the Cluster Reserve is set >1.

The output file details what would happen to the memory resources of each host if it were to fail. The output also details the remaining cluster capacity in the event of each host failure. When attempting to pin-point a time when a cluster may become over-committed, by simply increasing either the memory allocated or demand, on a VM, the increased value will be deducted from each of the "Remaining cluster capacity (GB)" calculated for each of the host failure scenario.

It is my intention to develop this and further PowerShell scripts, to aid everyone to check and understand the Cluster Reserve feature as well as the people that wrote it!

If you have previous experience with this setting and have developed your own method of calculating your Cluster Reserve, please share them...this is the only way everyone (who needs to) will learn.

(The attached script is intended to be a guide and a method of exposing the values used to calculate the Cluster Reserve result)

Over-Committed_ProofFull_withOutput_v1.2.zip

Simply copy and paste the below into notepad, save as a .ps1 and run it!

You will be prompted for the VM Name, press enter and shortly after you will be given the Host and Cluster name where the VM is currently located.

I have also created another version that allows you to query multiple VMM Management servers: [View:~/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-00-96-43/FindMyVM_5F00_MultipleVMMServers.txt:550:0]

Since creating this article I have developed the script further to allow it to be run remotely from any machine. You are required to enter your VMM Server FQDN in the script before it will work in your environment. When running the script you will be prompted for credentials, which will be used to connect to the VMM Server and the name of the VM you want to find: [View:~/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-00-96-43/FindMyVM_5F00_Remotely.txt:550:0]

The user needed to create a new volume copy a large number of .vhd files to it and then clone the volume, creating multiple disks to be added as separate CSVs. The copying of the data and the cloning of the disks appeared to work fine, even the adding of the new volumes and enabling them as CSVs worked perfect, in Failover Cluster Manager (FCM). The problem came when the user wanted to be able to attach the .vhds stored on the new volumes to existing Virtual Machines (VMs) using a PowerShell script which included VMM cmdlets. Whenever they attempted to run the script, there were advised that the new volumes do not exist. However we could see them in FCM and even browse/read/write to them via Explorer. However, when looking at the properties of the cluster in VMM the new volumes did not appear. Below is a couple of example screenshots of the experienced issue:

New volume appearing in FCM

Emtpy Shared Volume list in VMM, cluster properties

When analyzing the individual hosts and checking what storage they can see, interestingly we can see reference to the volume in FCM under the Hardware section on the host properties, example:

With no number of manual VMM refresh tasks fixing the issue, I simply removed the storage, formatted the disk and re-added to the cluster (using FCM). After performing these simple tasks, in my case, the disk appeared in VMM cluster properties:

After performing these tests and investigating the issue, I am still yet to find out why the disks did not appear in VMM cluster properties...however what I have learned is that VMM doesn't appear to like cluster disks that have been added to a cluster without being freshly formatted.

After fixing the errors when performing a Live (VSM) migration, we found that on the destination host/cluster duplicate VMs/Roles in Failover Cluster Manager were created. The following video demonstrates this behaviour:

[View:~/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-00-96-43/0825.clustertocluster_5F00_migration.avi:0:0]

This issue was acknowledged and now FIXED in VMM 2012 R2 Rollup 3 (https://support.microsoft.com/kb/2965414).

We investigated the possibilities of using Failover Cluster Manager to migrate the VMs/Roles between clusters, however in this instance that method did not fit, due to the method requiring that the old storage be mapped to the new hosts (not possible due to hardware restrictions), also perhaps more importantly this method would incur downtime!

In addition to the environment being a mature Hyper-V estate, we were also very fortunate that the customer had previously chosen to centrally manage their environment using SCVMM 2012 R2. Within VMM, since SP1, we have the ability to perform cluster-to-cluster Live Storage Migrations, also referred to as "Live (VSM)" during the wizard. (AMAZING, I know!). For more details see: https://technet.microsoft.com/en-us/library/jj628158.aspx

So using the Live (VSM) feature in VMM we attempted to migrate a single VM, which worked perfectly. We then set about migrating further machines, however after 3 successful migrations, the 4th FAILED!. We attempted the same migration shortly after and it COMPLETED! whilst working through the list of VMs we saw a number of RANDOM failures, all producing the following error (Error (12700)):

The same error may also appear when attempting a Storage Migration using the Hyper-V Manager console:

Whilst investigating the error messages, I found that the Live Migration authentication method was set to CredSSP, instead of Kerberos (see for details: https://technet.microsoft.com/en-us/library/jj134199.aspx). After configuring all hosts with Constrained Delegation, we no longer received the above error when performing Live (VSM) migrations, in VMM, between clusters.

Although the initial issue had been resolved, we were left with another....(see for details: https://blogs.technet.com/b/virtual-mite/archive/2014/09/11/duplicate-vms-created-on-destination-during-cluster-to-cluster-migration.aspx)

The final task list included 22 steps, split into 3 sections (Prerequisites, Implementation and Testing). The majority of the steps involved were environmentally specific and will not apply to all setups. So here are the generic tasks that we had to complete in order to achieve a successful DB move:

Prerequisites

1. Ensure you have a recent backup of the VMM Management servers - In our case the customer used LUN Cloning technology to achieve this.
2. Backup the VMM DB, using SQL Management Studio
3. Ensure the relevant permissions are configured, as per the VMM Installation Prerequisites (https://technet.microsoft.com/en-us/library/gg610592.aspx)
4. Restore the backed up DB to the destination SQL Server - if you have HA VMM Management servers, you are required to clear the HAVMMNAME and HAVMMNodes entries in the restored DB (see for details: https://blogs.technet.com/b/virtual-mite/archive/2014/02/13/reinstalling-clustered-scvmm-management-server-using-existing-sql-database.aspx)

Implementation

(If you have a standalone VMM Management server perform Steps 4 and 6)

1. Take the VMM cluster resource offline - In Failover Cluster Manager select the SCVMM highly available role and "Take Offline"
2. Stop the System Center Virtual Machine Manager service on the active node - This will stop VMM users logging into the console whilst maintenance is performed
3. Uninstall SCVMM from passive node(s) & reboot
4. Uninstall SCVMM from active node & reboot - During the uninstallation wizard select "Retain the database", as a precaution
5. Delete DNS and WINS entries, for the VMM Cluster Name
6. Install VMM on the first node, pointing to the restored DB on the destination SQL Server. After completion, install the latest rollup patches
7. Install VMM on the other node(s). After complete install the latest rollup patches

Testing

1. Login to the VMM Console and confirm all Resources/Infrastructure Servers/VM Templates/Profiles...are available, as they were when using the original source SQL Server

(Each VMM environment/IT Admin may have their own testing plan to confirm functionality)

The above steps will also apply, in most cases, when upgrading your VMM installation. However, you will need to perform additional tasks (example: upgrading the Vmmagent version on the hosts) to complete a successful upgrade.

The above output file was a result of the following command: Get-SCVirtualMachine | ft Name | export-csv C:\Output.txt

In order to overcome this issue I have discovered "LogWrite", by simply adding the following lines to the beginning of your script, you will be able to log any results/comments to the output file:

$Logfile = Read-Host "Enter the LogFile Path (example: C:\Output.txt)"
Function LogWrite
{
Param ([string]$logstring)

Add-content $Logfile -value $logstring
}

So, now using the above command I am able to create a script that will return the values I was originally attempting to capture.

Script used to generate the above Output.txt:

$Logfile = Read-Host "Enter the LogFile Path (example: C:\Output.txt)"
Function LogWrite
{
Param ([string]$logstring)
Add-content $Logfile -value $logstring
}
LogWrite "VMName"
ForEach ($VM in Get-SCVirtualMachine){
LogWrite $VM.Name
}

This is just another one of those simple PowerShell tricks that allow you to overcome a frustrating task.

Express Installation - This type of installation should only be used in test and development environments. An Express installation allows you to install all of the WAP roles on a single server. This method is NOT supported in a production environment

Distributed installation - This method requires you to install the WAP roles on separate servers. The entry level installation using this method requires 8 servers (physical or virtual). This method is supported in production. Below is a list of the 8 roles (7 of which are installed using the Web Platform Installer (WPI)) that make up an installation of WAP.

1. Microsoft SQL Server for Windows Azure Pack management databases
2. Admin API
3. Tenant API
4. Tenant public API
5. Admin authentication site
6. Tenant authentication site
7. Admin site (management portal for administrators)
8. Tenant site (management portal for tenants)

For more information see: https://technet.microsoft.com/en-us/library/dn296442.aspx

Although the above information is useful, the reason for writing this post was to add further publicity to an issue we experienced during the installation.

Once we had completed the installation, we registered SPF in WAP and created a User Account. We then continued to create a Plan for a Cloud that we had created in VMM. When it came to adding the Plan to the User Account, as a Subscription. The Subscription status remained in "Active (Out of Sync)", which meant that we were unable to deploy VMs using this account and subscription. After a bit of effort it was highlighted that in order to over come this issue, we had to install VMM 2012 R2 Rollup 1 (https://support.microsoft.com/kb/2904712) As part of the installation you must also ensure that you run the SQL query included in the KB, otherwise the problem will persist.