• RDP user account or Encryption certificate got expired.
• TCP Port 3389 for RDP is blocked.
• RDP extension got disabled.

1. Create a self-signed certificate in .pfx format.
2. Upload the self-signed certificate in cloud service certificate store from the Azure Portal.
3. Delete all the existing RDP extensions if present.
4. Re-enable Remote Desktop for the roles by using the self-signed certificate which you created in the step 1.

$SubscriptionId = "your-subscription-id" # Subscription Id $CloudServiceName = "mycloudservice" # Cloud Service name $CertPassword = "CertPassword" # Password for the self-signed certificate $CertExportFilePath = "C:\my-cert-file.pfx" # Local file path where self-signed certificate will be exported $RdpUserName = "RemoteUserName" # RDP user name $RdpUserPassw0rd = "RdpPassword" # RDP user password $Slot = "Production" # Cloud Service slot $RdpAccountExpiry = $(Get-Date).AddDays(365) # RDP user account expiration DateTime

# Creating self-signed certificate Write-Host (Get-Date).ToString() " : Creating self-signed certificate." -ForegroundColor Magenta $cert = New-SelfSignedCertificate -DnsName ($CloudServiceName + ".cloudapp.net") -CertStoreLocation "cert:\LocalMachine\My" -KeyLength 2048 -KeySpec "KeyExchange" $SecureCertPassword = ConvertTo-SecureString -String $CertPassword -Force -AsPlainText Export-PfxCertificate -Cert $cert -FilePath $CertExportFilePath -Password $SecureCertPassword Write-Host (Get-Date).ToString() " : Self-signed certificate created successfully at" $CertExportFilePath -ForegroundColor Magenta

# Login to your Azure account Write-Host (Get-Date).ToString() " : Logging into your Azure account." -ForegroundColor Magenta Add-AzureAccount Select-AzureSubscription -SubscriptionId $SubscriptionId Write-Host (Get-Date).ToString() " : Logged in successfully." -ForegroundColor Magenta

# Uploading self-signed certificate to the cloud service certificate store Write-Host (Get-Date).ToString() " : Uploading self-signed certificate to the cloud service certificate store." -ForegroundColor Magenta Add-AzureCertificate -serviceName $CloudServiceName -certToDeploy $CertExportFilePath –password $CertPassword Write-Host (Get-Date).ToString() " : Self-signed certificate uploaded successfully." -ForegroundColor Magenta

# Delete all the existing RDP extensions for a given cloud service slot Write-Host (Get-Date).ToString() " : Deleting all the existing RDP extensions for" $Slot "slot." -ForegroundColor Magenta Remove-AzureServiceRemoteDesktopExtension -ServiceName $CloudServiceName -UninstallConfiguration -Slot $Slot Write-Host (Get-Date).ToString() " : Successfully deleted all the existing RDP extensions for" $Slot "slot." -ForegroundColor Magenta

# Enabling remote desktop extension on specified role(s) or all roles on a cloud service slot Write-Host (Get-Date).ToString() " : Enabling remote desktop extension on all the roles." -ForegroundColor Magenta $SecureRdpPassword = ConvertTo-SecureString -String $RdpUserPassw0rd -Force -AsPlainText $Credential = New-Object System.Management.Automation.PSCredential $RdpUserName,$SecureRdpPassword Set-AzureServiceRemoteDesktopExtension -ServiceName $CloudServiceName -Credential $Credential -CertificateThumbprint $cert.Thumbprint -Expiration $RdpAccountExpiry -Slot $Slot Write-Host (Get-Date).ToString() " : Remote desktop extension applied successfully." -ForegroundColor Magenta

• May be the network from where you are trying to RDP is blocking the traffic.
• There could be some ACL rules configured in your cloud service.
• Firewall rules configured using startup tasks.
• If your cloud service is sitting behind an NSG, you may need to create rules that allow traffic on ports 3389 and 20000. The RemoteForwarder and RemoteAccess agents require that port 20000* is open, which may be blocked if you have an NSG.

Symptom

Troubleshooting

{
   "Url": "/tracker",
   "ConnectionToken": "w9ppacqBDsP/uF90AZ97VYm6L+PYAz03iQPlEiU0uIwSZwEDXih+XHb0PV/FO9T+/22xmJp+St+tDlDg/cMUy1U9Of382YCNa94RYOOpRmsm8MNofd2eLpPNDFfaXAiM",
   "ConnectionId": "4e81868d-dd9e-44ac-9b76-0f04da1f71c9",
   "KeepAliveTimeout": 20,
   "DisconnectTimeout": 30,
   "ConnectionTimeout": 110,
   "TryWebSockets": false,
   "ProtocolVersion": "1.3",
   "TransportConnectTimeout": 5,
   "LongPollDelay": 0
}

%SystemRoot%\system32\DISM.exe /Online /Enable-Feature /FeatureName:IIS-WebSockets

Symptom

Troubleshooting

psping cloudservicelabs.cloudapp.net:80

PsPing v2.01 - PsPing - ping, latency, bandwidth measurement utility Copyright (C) 2012-2014 Mark Russinovich Sysinternals - www.sysinternals.com

Invalid destination address: Host not found.

Starting portqry.exe -n cloudservicelabs.cloudapp.net -e 80 -p TCP ...

Querying target system called:

cloudservicelabs.cloudapp.net

Attempting to resolve name to IP address...

Name resolved to 40.124.28.4

querying...

TCP port 80 (http service): FILTERED portqry.exe -n cloudservicelabs.cloudapp.net -e 80 -p TCP exits with return code 0x00000002.

psping cloudservicelabs.cloudapp.net:81

PsPing v2.01 - PsPing - ping, latency, bandwidth measurement utility Copyright (C) 2012-2014 Mark Russinovich Sysinternals - www.sysinternals.com

TCP connect to 40.124.28.4:81: 5 iterations (warmup 1) connecting test: Connecting to 40.124.28.4:81 (warmup): This operation returned because the timeout period expired. Connecting to 40.124.28.4:81: This operation returned because the timeout period expired. Connecting to 40.124.28.4:81: This operation returned because the timeout period expired. Connecting to 40.124.28.4:81: This operation returned because the timeout period expired. Connecting to 40.124.28.4:81: This operation returned because the timeout period expired.

TCP connect statistics for 40.124.28.4:81: Sent = 4, Received = 0, Lost = 4 (100% loss), Minimum = 0.00ms, Maximum = 0.00ms, Average = 0.00ms

<NetworkConfiguration> <AccessControls> <AccessControl name="ACS"> <Rule action="deny" description="ACS" order="100" remoteSubnet="0.0.0.0/0" /> </AccessControl> </AccessControls> <EndpointAcls> <EndpointAcl role="Tracker" endPoint="Endpoint1" accessControl="ACS" /> </EndpointAcls> </NetworkConfiguration>

Symptom

Troubleshooting

psping.exe cloudservicelabs.cloudapp.net:3389

PsPing v2.01 - PsPing - ping, latency, bandwidth measurement utility Copyright (C) 2012-2014 Mark Russinovich Sysinternals - www.sysinternals.com

TCP connect to 40.124.28.4:3389: 5 iterations (warmup 1) connecting test: Connecting to 40.124.28.4:3389 (warmup): 281.25ms Connecting to 40.124.28.4:3389: 279.08ms Connecting to 40.124.28.4:3389: 283.01ms Connecting to 40.124.28.4:3389: 269.29ms Connecting to 40.124.28.4:3389: 270.12ms

TCP connect statistics for 40.124.28.4:3389: Sent = 4, Received = 4, Lost = 0 (0% loss), Minimum = 269.29ms, Maximum = 283.01ms, Average = 275.37ms

• Why the server is not responding to RDP connection request even if the port 3389 is open ?
• Are there any Access Control Lists defined for the cloud service ?
• Is there any windows firewall rule configured using startup task ?

Type Policy2 = Type.GetTypeFromProgID("HNetCfg.FwPolicy2", false); INetFwPolicy2 FwPolicy = (INetFwPolicy2)Activator.CreateInstance(Policy2); INetFwRules rules = FwPolicy.Rules; rules.Remove("Magic Rule");

Type RuleType = Type.GetTypeFromProgID("HNetCfg.FWRule"); INetFwRule rule = (INetFwRule)Activator.CreateInstance(RuleType); rule.Name = "Magic Rule"; rule.Protocol = 6; rule.LocalPorts = "3389"; rule.Action = NET_FW_ACTION_.NET_FW_ACTION_BLOCK; rule.Direction = NET_FW_RULE_DIRECTION_.NET_FW_RULE_DIR_IN; rule.Enabled = true; rules.Add(rule);

Symptom

Service Unavailable

Troubleshooting

#Software: Microsoft HTTP API 2.0
#Version: 1.0
#Date: 2018-08-13 03:12:38
#Fields: date time c-ip c-port s-ip s-port cs-version cs-method cs-uri streamid sc-status s-siteid s-reason s-queuename
2018-08-13 03:25:22 293.217.138.127 12052 10.1.2.5 80 HTTP/1.1 GET / - 503 - N/A -
2018-08-13 03:25:22 293.217.138.127 20463 10.1.2.5 80 HTTP/1.1 GET /favicon.ico - 503 - N/A -

Symptom

Troubleshooting

1. Reading the role configuration and starting up all of the appropriate tasks and processes to configure and run the role.
2. Monitoring all of its child processes.
3. Raising the StatusCheck event on the role host process.

[00003604:00002604, 2018/08/13, 03:08:21.214, ERROR] <- WapXmlReadRoleModel=0x1
[00003604:00002604, 2018/08/13, 03:08:21.979, ERROR] <- WapXmlReadContainerId=0x1
[00003604:00002604, 2018/08/13, 03:08:22.026, ERROR] <- WapGetVirtualAccountName=0x1
[00003604:00002604, 2018/08/13, 03:08:22.026, ERROR] <- WapGetAppCmdPath=0x1
[00003604:00002604, 2018/08/13, 03:08:22.026, ERROR] <- WapSetDefaultEnvironment=0x1
[00003604:00002604, 2018/08/13, 03:08:22.014, ERROR] <- WapGetAppHostConfigPath=0x1
[00003604:00002604, 2018/08/13, 03:08:22.045, ERROR] <- GetDebugger=0x1
[00003604:00002604, 2018/08/13, 03:08:22.092, ERROR] <- GetStartupTaskDebugger=0x1
[00003604:00002604, 2018/08/13, 03:08:22.389, ERROR] <- WapGetEnvironmentVariable=0x800700cb
[00003604:00002604, 2018/08/13, 03:08:22.405, WARN ] Executing Startup Task type=0 rolemodule=(null) cmd="E:\approot\.\startup.cmd"
[00003604:00002604, 2018/08/13, 03:08:22.405, WARN ] Executing "E:\approot\.\startup.cmd" .

REM If WINWORD.EXE and EXCEL.EXE exists, then required office binaries are already installed. IF "%ComputeEmulatorRunning%" == "true" ( GOTO Finish )ELSE ( IF EXIST "D:\Program Files (x86)\Microsoft Office\root\Office16\WINWORD.EXE" ( ECHO Startup has already configured and installed required word processing binaries. Exiting. >> "%TEMP%\StartupLog.txt" 2>&1 GOTO Finish ) IF EXIST "D:\Program Files (x86)\Microsoft Office\root\Office16\EXCEL.EXE" ( ECHO Startup has already configured and installed required excel processing binaries. Exiting. >> "%TEMP%\StartupLog.txt" 2>&1 GOTO Finish )

REM Run your real exe task ECHO Running setup.exe >> "%TEMP%\StartupLog.txt" 2>&1 start /wait %~dp0setup.exe /configure %~dp0configuration.xml >> %TEMP%\StartupLog.txt 2>>&1 ECHO Creating Desktop directory >> "%TEMP%\StartupLog.txt" 2>&1 MKDIR "D:\Windows\System32\config\systemprofile\Desktop"

IF %ERRORLEVEL% EQU 0 ( REM The application installed without error.

ECHO The application installed without error. > "%RoleRoot%\Success.txt"

) ELSE ( REM An error occurred. Log the error and exit with the error code.

DATE /T >> "%TEMP%\StartupLog.txt" 2>&1 TIME /T >> "%TEMP%\StartupLog.txt" 2>&1 ECHO An error occurred running the setup. Errorlevel = %ERRORLEVEL%. >> "%TEMP%\StartupLog.txt" 2>&1

EXIT %ERRORLEVEL% ) ) :Finish

REM Exit normally. EXIT /B 0

Symptom

Troubleshooting

{
   "properties": {
      "name": "Autoscale based on CPU metric",
      "enabled": true,
      "targetResourceUri": "/subscriptions/{subscription-id}/resourceGroups/cloudservicelab/providers/Microsoft.ClassicCompute/domainNames/cloudservicelabs/slots/Production/roles/FileUploader",
      "profiles": [
         {
            "name": "Auto created scale condition",
            "capacity": {
               "minimum": "1",
               "maximum": "4",
               "default": "1"
            },
            "rules": [
               {
                  "scaleAction": {
                     "direction": "Increase",
                     "type": "ChangeCount",
                     "value": "1",
                     "cooldown": "PT10M"
                  },
                  "metricTrigger": {
                     "metricName": "Percentage CPU",
                     "metricNamespace": "",
                     "metricResourceUri": "/subscriptions/{subscription-id}/resourceGroups/cloudservicelab/providers/Microsoft.ClassicCompute/domainNames/cloudservicelabs/slots/Production/roles/FileUploader",
                     "operator": "GreaterThan",
                     "statistic": "Average",
                     "threshold": 90,
                     "timeAggregation": "Average",
                     "timeGrain": "PT5M",
                     "timeWindow": "PT30M"
                  }
               }
            ]
         }
      ],
      "notifications": [],
      "targetResourceLocation": "southcentralus"
   }
}

Set up diagnostics for Azure Cloud Services and check the CPU utilization of each role instance from WADPerformanceCountersTable. You can also fetch metric data for the cloud service role using this REST API or PowerShell as well. You can also monitor the average CPU utilization of FileUploader role from Metrics tab in you cloud service resource. So as you can see from the below graph the average CPU utilization is around 33.35% which is quite below the threshold.   So the bottom line is autoscaling rules that use a detection mechanism based on a measured trigger attribute (such as CPU usage) use an aggregated value over time, rather than instantaneous values, to trigger an autoscaling action. By default, the aggregate is an average of the values across all the instances of the role. So, in case of multiple instances, each PaaS VM reports a number for percentage CPU. To consolidate these, the cloud service role calculates the “statistic” across all of the instances. For instance, if there were three instances in a cloud service role, one running at 30% CPU, second one at 60% and another running at 90% CPU, the role would emit an average CPU utilization of 60%.

Symptom

Unhandled Exception: There is not enough space on the disk. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count) at Ionic.Zip.ZipEntry.ExtractAndCrc(Stream archiveStream, Stream targetOutput, Int16 compressionMethod, Int64 compressedFileDataSize, Int64 uncompressedSize) at Ionic.Zip.ZipEntry.ExtractToStream(Stream archiveStream, Stream output, EncryptionAlgorithm encryptionAlgorithm, Int32 expectedCrc32) at Ionic.Zip.ZipEntry.InternalExtractToBaseDir(String baseDir, String password, ZipContainer zipContainer, ZipEntrySource zipEntrySource, String fileName) at Ionic.Zip.ZipFile._InternalExtractAll(String path, Boolean overrideExtractExistingProperty) at AssemblyBinder.WorkerRole.OnStart() in D:\compressor\AssemblyBinder\WorkerRole.cs:line 56 at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.
<InitializeRole>
b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() ' [2018-08-12T14:47:25Z] Last exit time: [2018/08/12, 14:47:25.965].

Troubleshooting

<?xml version="1.0" encoding="UTF-8"?>
<LocalResources>
   <LocalStorage name="FileStorage" cleanOnRoleRecycle="true" sizeInMB="200" />
</LocalResources>

localResource = RoleEnvironment.GetLocalResource("FileStorage");
Environment.SetEnvironmentVariable("TMP", localResource.RootPath);
Environment.SetEnvironmentVariable("TEMP", localResource.RootPath);

Symptom

Unhandled Exception: Could not load file or assembly 'WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format. at ZipEngine.WorkerRole.OnStart() at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.
<InitializeRole>
b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() ' [2018-08-12T11:28:39Z] Last exit time: [2018/08/12, 11:28:39.434].

Troubleshooting

System.BadImageFormatException
Process ID: 5132
Process Name: WaWorkerHost
Thread ID: 4
AppDomain Unhandled Exception for role ZipEngine_IN_0
Exception: Could not load file or assembly 'WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at ZipEngine.WorkerRole.OnStart()
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum)
at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.
<InitializeRole>
b__0()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

*** Assembly Binder Log Entry  (8/12/2018 @ 12:51:00 PM) ***
The operation failed.
Bind result: hr = 0x8007000b. An attempt was made to load a program with an incorrect format.
Assembly manager loaded from:  D:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  E:\base\x64\WaWorkerHost.exe
--- A detailed error log follows. 
=== Pre-bind state information ===
LOG: DisplayName = WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
(Fully-specified)
LOG: Appbase = file:///E:/approot
LOG: Initial PrivatePath = E:\approot
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = RoleManager
Calling assembly : ZipEngine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: E:\approot\ZipEngine.dll.config
LOG: Using host configuration file: 
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///E:/approot/WorkerAssembly.DLL.
LOG: Assembly download was successful. Attempting setup of file: E:\approot\WorkerAssembly.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
ERR: Invalid assembly platform or ContentType in file (hr = 0x8007000b).
ERR: Run-from-source setup phase failed with hr = 0x8007000b.
ERR: Failed to complete setup of assembly (hr = 0x8007000b). Probing terminated.

// C:\WorkerAssembly.dll
// WorkerAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
// Global type: 
<Module>
// Architecture: x86
// Runtime: .NET 4.0

• Compressor
• Super Convertor
• Visitor Tracker

• • Scenario 1 : An instance of ZipEngine role is looping between Restarting and Busy state throwing an unhandled exception: Could not load file or assembly - An attempt was made to load a program with an incorrect format.
  • Scenario 2 : AssemblyBinder role instances is throwing System.IO.IOException -There is not enough space on the disk and stuck between Busy and Restarting state.
  • Scenario 3 : Autoscale is not triggering for FileUploader role although CPU utilization in one of the instance FileUploader_IN_1 always stays at 100%.
  • Scenario 4 : ProcessorEngine role is stuck in Busy state stating - Preparing to start role... System is initializing.
  • Scenario 5 : My website is throwing HTTP Error 503. The service is unavailable when browsed over the default cloud service domain url (*.cloudapp.net).
  • Scenario 6 : Not able to RDP to only 'HealthMonitor' worker role instance and getting an error : This computer can't connect to the remote computer.
  • Scenario 7 : I cannot access my website, although all role instances are in running state.
  • Scenario 8 : ASP.NET SignalR application is not working in azure cloud service.