Error 1: Exercise "FileServer" failed after the maximum number of retries. Restarting OnStart.

Assessment: The exercise means “health check” and the publisher is checking if it has access to the file server using FileShareOwner credential.

Action Plan: Check connectivity from the management server to the file server and from the publisher to the file server.

To manually check connectivity, you can do the following:
runas /user:FileShareOwner /netonly cmd
dir \Websites">\\<file-server-name>\Websites

Error 2: Failed to run operation 'Repair'.  Operation failed to complete. Connecting to remote server '<server name>' failed with the following error message : WinRM cannot complete the operation.

Assessment: Controller is unable to connect to remote server over port 5985.

Action Plan: Check Firewall settings and ports as per given article:

Ports Used and Network Topology in Windows Azure Pack: Web Sites
https://technet.microsoft.com/en-us/library/dn594513.aspx

Error 3: Waiting for worker data service credentials. Attempt number is 29220.

Assessment: Controller is unable to connect to web worker role server.

Action Plan: Check Firewall settings and ports as per given article:

Ports Used and Network Topology in Windows Azure Pack: Web Sites
https://technet.microsoft.com/en-us/library/dn594513.aspx

The symptoms seen in this scenario also are the same as we had seen in Part 1, but the underlying issue is different.

AzurePack: Unable to create database when using SQL High Availability group (AlwaysOn)- Part 1

We have setup WAP environment and configured a SQL resource provider. WAPSQLCL is the SQL listener that we have added to the AlwaysOn group.
When a tenant is created a new database, it fails with following message logged in tenant portal event logs:

Log Name:      Microsoft-WindowsAzurePack-MgmtSvc-TenantSite/Operational
Source:        Microsoft-WindowsAzurePack-MgmtSvc-TenantSite
Date:          xxxx
Event ID:      12
Task Category: (65522)
Level:         Error
Keywords:      None
User:          IIS APPPOOL\MgmtSvc-TenantSite
Computer:      xxxx
Description:
Error:JsonException: Unable to create the database 'test'. Please try again later. If the problem persists, please contact support.

Assessment:

In event logs we could see the detailed error:

Database 'test' could not be created on always on listener 'WAPSQLCL' since it restoration failed from network file share '\\Fileshare\Test' on secondary replica: WAPDB02 with following exception: System.Data.SqlClient.SqlException (0x80131904): Login failed for user 'SqlResourceProviderLogin'.

   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

In SQL server error logs we could see the error been logged as:

Login failed for user “sqlResourceProviderLogin”. Reason: password did not match that for the login provided.

Solution:

To sync the password of sqlResourceProviderLogin on both the SQL server nodes, select "Repair connection" option for SQL listener on Admin portal.

Import-Module 'MgmtSvcAdmin'
 
# environment settings
$AuthSiteURL= 'https://waptest:30072'
$AdminSiteURL= 'https://waptest:30004'
$ClientRealmUrl = 'https://azureservices/AdminSite'
 
# credentials and token
$password = ConvertTo-SecureString "password" -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential ("domain\username",$password)
$token = Get-MgmtSvcToken -Type Windows -AuthenticationSite $AuthSiteURL -ClientRealm $ClientRealmUrl -User $credential -DisableCertificateValidation
 
# plan / service settings
$PlanDisplayName = 'PlanName'
$planServiceName = 'systemcenter'
$serviceDisplayName = 'Virtual Machine Clouds'
$serviceInstanceDisplayName = 'Virtual Machine Clouds'
 
# quota settings (for System Center, these values are complex and depend upon the environment; one example shown for each)
$quotaSettingKey1 = "Actions"
$quotaSettingValue1 = '<Actions><Stamp Id="bbb46507-5cdd-4d23-830b-080ca2d5ff29"><Action>Checkpoint</Action><Action>CheckpointRestoreOnly</Action><Action>Save</Action><Action>Store</Action>

<Action>RemoteConnect</Action><Action>Author</Action><Action>Create</Action><Action>CreateFromVHDOrTemplate</Action>

<Action>AllowLocalAdmin</Action><Action>Start</Action><Action>Stop</Action><Action>PauseAndResume</Action><Action>Shutdown</Action>

<Action>Remove</Action><Action MaximumNetwork="" MaximumMemberNetwork="" MaximumBandwidthIn="" MaximumBandwidthOut="" MaximumVPNConnection="99" MaximumMemberVPNConnection="99">AuthorVMNetwork</Action></Stamp></Actions>'
 
$quotaSettingKey2 = "Clouds"
$quotaSettingValue2 = '<Clouds><Cloud Id="2b5cc12d-892b-45af-adcf-8f0f6df6eb35" StampId="bbb46507-5cdd-4d23-830b-080ca2d5ff29"><Quota><RoleVMCount></RoleVMCount><MemberVMCount></MemberVMCount><RoleCPUCount></RoleCPUCount>

<MemberCPUCount></MemberCPUCount><RoleMemoryMB></RoleMemoryMB><MemberMemoryMB></MemberMemoryMB>

<RoleStorageGB></RoleStorageGB><MemberStorageGB></MemberStorageGB></Quota></Cloud></Clouds>'
 
$quotaSettingKey3 = "VmResources"
$quotaSettingValue3 = '<Resources><HardwareProfile Id = "326fbb68-7ee7-4459-bc84-047bff491512" StampId="bbb46507-5cdd-4d23-830b-080ca2d5ff29"/><VmTemplate Id = "b88c24e0-06a1-4ed0-bfbc-22e2949002ac" StampId="bbb46507-5cdd-4d23-830b-080ca2d5ff29"/><GalleryItem Name = "LarysaResDef" Version = "3.0.0.0" Publisher = "UnspecifiedPublisher" StampId="bbb46507-5cdd-4d23-830b-080ca2d5ff29"/></Resources>'
 
$quotaSettingKey4 = "Networks"
$quotaSettingValue4 = '<Networks><Network Id = "354a27eb-2705-4073-9dbb-52cec271f839" StampId="bbb46507-5cdd-4d23-830b-080ca2d5ff29"/></Networks>'
 
$quotaSettingKey4 = "CustomSettings"
$quotaSettingValue4 = '<CustomSettings></CustomSettings>'
 
# Add a new public Management Service plan
$plan = Add-MgmtSvcPlan $AdminSiteURL $token -DisableCertificateValidation -DisplayName $PlanDisplayName -State Public
 
# Retrieve the System Center resource provider config data (the System Center resource provider should already exist in the Management Service node)
$systemCenterRP = Get-MgmtSvcResourceProvider -AdminUri $AdminSiteURL -token $token -disablecertificatevalidation -name $planServiceName
 
# Add the System Center service to the plan created
Add-MgmtSvcPlanService $AdminSiteURL $token -DisableCertificateValidation -PlanId $plan.Id -ServiceName $planServiceName -InstanceId $systemCenterRP.InstanceId
 
# Add Quotas to the plan created
$QuotaList = New-MgmtSvcQuotaList
 
$quota = Add-MgmtSvcListQuota -QuotaList $QuotaList -ServiceName $planServiceName -ServiceInstanceId $systemCenterRP.InstanceId
$quota.ServiceDisplayName = $serviceDisplayName
$quota.ServiceInstanceDisplayName = $serviceInstanceDisplayName
 
# add each of the key:value quota settings to the quota (each saved into settings variable for debug inspection or logging)
$setting1 = Add-MgmtSvcQuotaSetting -Quota $quota -Key $quotaSettingKey1 -Value $quotaSettingValue1
$setting2 = Add-MgmtSvcQuotaSetting -Quota $quota -Key $quotaSettingKey2 -Value $quotaSettingValue2
$setting3 = Add-MgmtSvcQuotaSetting -Quota $quota -Key $quotaSettingKey3 -Value $quotaSettingValue3
$setting4 = Add-MgmtSvcQuotaSetting -Quota $quota -Key $quotaSettingKey4 -Value $quotaSettingValue4
 
Update-MgmtSvcPlanQuota $AdminSiteURL $token -DisableCertificateValidation -PlanId $plan.Id -QuotaList $QuotaList -Verbose

In case you have moved your WAP database to a different server or need to reset password for a SQL login, we would need the connection string from web.config of AzuPack components.

The web.config files are encrypted so they should be decrypted first. You can do this through the following PowerShell code:

Get-MgmtSvcNamespace | %{
Unprotect-MgmtSvcConfiguration -Namespace $_
}

This code will unencrypt all web.config files associated with the namespaces installed on the local server. When this has run you can go to IIS manager and reconfigure the connection strings.

When every connection string is adjusted, you can run the following PowerShell code to encrypt the web.config files again:

Get-MgmtSvcNamespace | %{
Protect-MgmtSvcConfiguration -Namespace $_
}

After installing a rollup update on AzurePack, you might sometimes get into an issue where the version of WAP component doesn’t match for database schema version. Most commonly seen error message or symptom of this behavior as seen in event logs is:

System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure 'mp.UpdatePlan_vx_xx'.

Stored procedure name in the error message would differ based on what operation you are performing on the portal. But the underlying issue is that database schema is not updated to the version of WAP components that we have on the servers.

So first we need to verify if there is indeed a version mismatch.

You can determine the versions of your Windows Azure Pack installations with the Get-WmiObject Windows PowerShell command. Run the following command on each computer that has a Windows Azure Pack installed, for example:

Get-WmiObject -Class Win32_Product | Where-Object { $_.Name -like "Windows Azure Pack *" } | Format-Table -AutoSize

To determine the versions of your Windows Azure Pack databases, run the following query in SQL Server Management Studio on the SQL Server instance that hosts the Windows Azure Pack databases:

-- WAP database versions
SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.Config' AS [Database],
N'Config' AS [Schema], *
FROM [Microsoft.MgmtSvc.Config].[Config].[Version]
UNION

SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.PortalConfigStore' AS [Database],
N'Config' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.PortalConfigStore].[Config].[Version]
UNION
SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.PortalConfigStore' AS [Database],
N'PortalAspNet' AS [Schema],
N'' AS [Version], [CompatibleSchemaVersion] AS [Major], 0 AS [Minor], 0 AS [Build], 0 AS [Revision], N'' AS [VersionInfo]
FROM [Microsoft.MgmtSvc.PortalConfigStore].[dbo].[aspnet_SchemaVersions]
WHERE [Feature] = N'membership'
UNION

SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.Store' AS [Database],
N'Config' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.Store].[Config].[Version]
UNION
SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.Store' AS [Database],
N'Management' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.Store].[mp].[Version]
UNION

SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.Usage' AS [Database],
N'Usage' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.Usage].[usage].[Version]
UNION

SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.WebAppGallery' AS [Database],
N'WebAppGallery' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.WebAppGallery].[WebAppGallery].[Version]
UNION

SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.SQLServer' AS [Database],
N'SQLServer' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.SQLServer].[SqlServer].[Version]
UNION

SELECT SERVERPROPERTY(N'ServerName') AS [Server],
N'Microsoft.MgmtSvc.MySQL' AS [Database],
N'MySQL' AS [Schema],
[Version], [Major], [Minor], [Build], [Revision], [VersionInfo]
FROM [Microsoft.MgmtSvc.MySQL].[MySql].[Version]

If Windows Azure Pack component version is greater than the database version, run the following script as administrator on the servers that contain the MgmtSvc-PowerShellAPI module. These servers include the Usage, Web App Gallery, SQL Server, and MySQL extensions.

Note that this script may need modification for some distributed installations. In particular, the schemas for Usage, WebAppGallery, SQLServer, and MySQL are installed with those components so those portions of the script would need to run on the computers with those components installed.

Caution: Take a backup of your databases before running the script.

# Update-WapDatabases
Import-Module -Name MgmtSvcConfig

function New-SqlConnectionString([string]$masterConnectionString, [string]$database)
{
    $builder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder($masterConnectionString)
    $builder.Database = $database
    return $builder.ConnectionString
}
function Get-WapSchemas([string]$database)
{
    switch ($database)
    {
        "Microsoft.MgmtSvc.Config"            { @("Config") }
        "Microsoft.MgmtSvc.MySQL"             { @("MySQL") }
        "Microsoft.MgmtSvc.PortalConfigStore" { @("Config","PortalAspNet","PortalNotification") }
        "Microsoft.MgmtSvc.SQLServer"         { @("SQLServer") }
        "Microsoft.MgmtSvc.Store"             { @("Config","Management") }
        "Microsoft.MgmtSvc.Usage"             { @("Usage") }
        "Microsoft.MgmtSvc.WebAppGallery"     { @("WebAppGallery") }
        default { throw New-Object System.ArgumentOutOfRangeException($database) }
    }
}

# Prompt for the SQL Server name:
$sName = Read-Host "Specify the name of the SQL Server that hosts the Windows Azure Pack databases."

$wapMasterConnectionString = "Server=" + $sName + ";Database=master;Integrated Security=True"

$wapDatabaseNames = (Get-MgmtSvcDefaultDatabaseName).DefaultDatabaseName
foreach ($wapDatabaseName in $wapDatabaseNames)
{
    $wapConnectionString = New-SqlConnectionString -masterConnectionString $wapMasterConnectionString -database $wapDatabaseName
    Write-Verbose -Message "Connection string: $wapConnectionString" -Verbose

    $wapSchemas = Get-WapSchemas -database $wapDatabaseName
    foreach ($wapSchema in $wapSchemas)
    {
        $wapSchema = Get-MgmtSvcSchema -Schema $wapSchema
        if ($wapSchema)
        {
            Write-Verbose -Message "BEGIN UPDATE database '$wapDatabaseName' schema '$wapSchema'." -Verbose
            Install-MgmtSvcDatabase -ConnectionString $wapConnectionString -Schema $wapSchema
            Write-Verbose -Message "END UPDATE database '$wapDatabaseName' schema '$wapSchema'." -Verbose
            $version = Test-MgmtSvcDatabase -ConnectionString $wapConnectionString -Schema $wapSchema
            Write-Output "Version: database '$wapDatabaseName' schema '$wapSchema' version $version"
        }
    }
}

Reference: https://technet.microsoft.com/en-us/library/dn747884.aspx

One of the common issues we get after setting up AzurePack is while trying to access Admin portal and getting HTTP 500. Moreover issue would be limited to admin portal only and tenant portal might work fine.

AzurePack admin portal uses Windows authentication whereas tenant portal used forms authentication, hence the difference.

The admin user account with which we are trying to access the admin portal might be a system admin or domain admin account that would have large number of group memberships. With the increase in number of group memberships, the size of auth token also increases.

This could be verified by looking at a network trace or simply run fiddler on the client machine and check the header size.

Suggested way of fixing such issues is to reduce the number of group memberships. But that might not be always possible.

Another way to fix this issue is to increase the allowed limit on the web server. We have to increase MaxRequestBytes and MaxFieldLength values on all servers where is Windows Authentication Role installed:

Key path: SYSTEM\CurrentControlSet\Services\HTTP\Parameters
Value name: MaxRequestBytes
Value type: REG_DWORD
Value data: 65534

Key path: SYSTEM\CurrentControlSet\Services\HTTP\Parameters
Value name: MaxFieldLength
Value type: REG_DWORD
Value data: 65534

Reference:

Problems with Kerberos authentication when a user belongs to many groups
https://support.microsoft.com/en-us/kb/327825

Troubleshooting tricks from my wiki

1. When creating a new website from Gallery, if Gallery items are not populated, then check if these tables are populated:

[Microsoft.MgmtSvc.WebAppGallery].[WebAppGallery].[MarketplaceItemProperties]

[Microsoft.MgmtSvc.WebAppGallery].[WebAppGallery].[MarketplaceItems]

2. Also check the events on the server on which this service is hosted --> MgmtSvc-WebAppGallery

3. If the gallery items are populated but still unable to publish the website from Gallery, then check if you are able to ping the Publishing server or if Publishing servers are load balanced. Load balancer can be bypassed to rule out any issues with it.

4. Try creating a blank website and download its publishing profile. Using this Publishing profile in WebMartrix, try to publish a web site from Gallery items. WebMatrix gives option to deploy using WebDeploy or FTP. Try both options and check the errors.

Hope this helps!

Quite often I have come across situations where ADFS trust was broken for admin or the tenant portal. And the only option we had was to rerun the trust settings again.

To re-run the trust between all components, we will follow the steps mentioned in this article:

https://technet.microsoft.com/en-us/library/dn457746.aspx

Following steps are from my wiki to perform ADFS re-run:

Step 1: Configure the Admin\Tenant management portals to trust AD FS

On Admin Portal server:

Set-MgmtSvcRelyingPartySettings –Target Admin –MetadataEndpoint https://<fqdn>/FederationMetadata/2007-06/FederationMetadata.xml -ConnectionString $ConnectionString

On Tenant Portal server:

Set-MgmtSvcRelyingPartySettings –Target Tenant –MetadataEndpoint https://<fqdn>/FederationMetadata/2007-06/FederationMetadata.xml -ConnectionString $ConnectionString

Step 2: Add users to have access to the management portal for administrators

On Admin API server:

$adminuser = 'domainuser1@mydomain.com'

$dbServer = 'AzurePack-sql'

$dbUsername = 'sa'

$dbPassword = 'SQL_Password'

$connectionString = [string]::Format('Server= {0} ;Initial Catalog=Microsoft.MgmtSvc.Store;User Id={1};Password={2};',$dbServer, $dbUsername, $dbPassword)

Add-MgmtSvcAdminUser -Principal $adminuser -ConnectionString $connectionstring

Step 3: Configure the Admin\Tenant authentication site to trust AD FS

On WindowsAuth Server:

Set-MgmtSvcIdentityProviderSettings –Target Windows –MetadataEndpoint https://<fqdn>/federationmetadata/2007-06/federationmetadata.xml -ConnectionString $connectionstring

On TenantAuth Server:

Set-MgmtSvcIdentityProviderSettings -Target Membership -MetadataEndpoint https://< fqdn>/FederationMetadata/2007-06/FederationMetadata.xml -ConnectionString $connectionstring

Step 4: Configure AD FS to trust the management portals

a. Locate the configure-adfs.ps1 configuration script that is installed with Windows Azure Pack in C:\Program Files\Management Service\MgmtSvc-PowerShellAPI\Samples\Authentication\.

b. Run the configure-adfs.ps1 script on the machine where AD FS is installed.

$tenantSite = 'tenant-AzurePack.contoso.com:30081'

$adminSite = 'admin-AzurePack.contoso.com:30091'

$authSite = 'auth-AzurePack.contoso.com:30071'

& "C:\Program Files\Management Service\MgmtSvc-PowerShellAPI\Samples\configure-adfs.ps1" ` –identityProviderMetadataEndpoint "https://$authSite/federationmetadata/2007-06/federationmetadata.xml" ` -tenantRelyingPartyMetadataEndpoint "https://$tenantSite/federationmetadata/2007-06/federationmetadata.xml" ` -adminRelyingPartyMetadataEndpoint "https://$adminSite/federationmetadata/2007-06/federationmetadata.xml" `

Appendix: How to break ADFS tenant trust

Note: Take a backup of Microsoft.MgmtSvc.Store database before breaking ADFS trust

Set-MgmtSvcRelyingPartySettings -Target Tenant -MetadataEndpoint https://wapportal:30071/FederationMetadata/2007-06/FederationMetadata.xml -ConnectionString $portalConfigStoreConnectionString -DisableCertificateValidation 

Set-MgmtSvcIdentityProviderSettings -Target Membership -MetadataEndpoint https://wapportal:30081/FederationMetadata/2007-06/FederationMetadata.xml -ConnectionString $portalConfigStoreConnectionString -DisableCertificateVaLidation

https://blogs.technet.com/b/privatecloud/archive/2014/02/11/custom-resource-providers-in-windows-azure-pack-deploying-the-hello-world-sample.aspx

Hope this helps!

To remove a resource provider from WAP portal like websites resource provider or SPF

1. Go to database to get the instance ID from [Microsoft.MgmtSvc.Store].[MP].[Resourceproviders]
2. Open PowerShell window as Run as administrator
3. Run Get-Help remove-MgmtSvcResourceProviderConfiguration -details
4. Remove-MgmtSvcResourceProviderConfiguration -Name <string> -Instancename <string>

Hope this helps!