Tim's blog

How to Add the OMS Client to a VM Scale Set

Tim Omta — Mon, 09 Apr 2018 12:03:12 GMT

I was asked by a customer to add the OMS client to an Azure VM ScaleSet (VMSS). This is not documented directly for a VMSS, though it is for Azure VMs. After some asking around and some experimentation, I was able to successfully deploy the OMS agent to the VMSS using the following PowerShell:

select-azurermsubscription -subscriptionid ‘your subscription id’
$PublicSettings = @{"workspaceId" = "your oms workspace id"}
$ProtectedSettings = @{"workspaceKey" = "your big base64 oms key"}

# Get information about the scale set
$vmss = Get-AzureRmVmss -ResourceGroupName 'VMSSRESOURCEGROUP' `
                        -VMScaleSetName 'VMSSNAME'

Add-AzureRmVmssExtension `
   -VirtualMachineScaleSet $vmss `
   -Name "Microsoft.EnterpriseCloud.Monitoring" `
   -Publisher "Microsoft.EnterpriseCloud.Monitoring" `
   -Type "MicrosoftMonitoringAgent" `
   -TypeHandlerVersion 1.0 `
   -AutoUpgradeMinorVersion $true `
   -Setting $PublicSettings `
   -ProtectedSetting $ProtectedSettings 

# Update the scale set and apply the Custom Script Extension to the VM instances
Update-AzureRmVmss `
              -ResourceGroupName $vmss.ResourceGroupName `
              -Name $vmss.Name `
              -VirtualMachineScaleSet $vmss

# Only needed for manual update VMSS – warning tells them all to update, so modify to suit
$jobs=@()
Get-AzureRmVmssVM -ResourceGroupName $vmss.ResourceGroupName -VMScaleSetName $vmss.Name | foreach {
    $jobs+=Update-AzureRmVmssInstance -ResourceGroupName $vmss.ResourceGroupName -Name $vmss.Name -InstanceId $_.InstanceId -AsJob
}

$jobs | Wait-Job
$jobs | Receive-Job

References Azure VMSS Azure Operations Management Suite (OMS) Azure VM OMS Agent

Stand Alone Service Fabric Error=EndpointProviderPortRangeExhausted

Tim Omta — Thu, 14 Dec 2017 15:23:06 GMT

I ran into an issue with a customer wherein they had set up an on-premise, or stand-alone, Service Fabric cluster. They had deployed several services to the cluster and some of the services would seem to randomly fail to start. The only way to see some information about the errors was to take a look in the Application Event logs on the nodes.

We observed errors that looked like this:

End(ActivateServicePackageInstance): AppId=mtsDeviceServiceType_App0, AppVersion=1.0 ServicePackageName=ContainerActorPkg, ServicePackageActivationContext=, ServicePackageVersionInstance=1.0:1.0:131575929604768573, Error=EndpointProviderPortRangeExhausted

It was odd to me because I had never seen this before in clusters hosted in Azure. After speaking with support, it turns out that stand-alone clusters have a much lower ephemeral port range of 30 ports by default. An Azure cluster is set much higher, which is why I had never seen the error there.

In this case the ephemeral port range was being used up on a first-come-first-served basis by whichever service started and got a port first, so the erroring applications looked random.

To correct the error, increase the # of application ports by modifying the standalone cluster templates (like ClusterConfig.X509.MultiMachine.json) to allocate a larger range:

"nodeTypes": [
{
"name": "NodeType0",
...
"applicationPorts": {
"startPort": "20001",
"endPort": "20031" ß use a higher number giving you a larger range
},
}

References

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-creation-for-windows-server

Where Did Application Insights Put my Performance Counter Data?

Tim Omta — Thu, 30 Nov 2017 12:21:31 GMT

I ran into an issue finding performance counter values I had pushed to Application Insights and wanted to note it to save others some time.

You can configure Azure Diagnostics (WAD) to push diagnostic and performance data to Application Insights:

https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/azure-diagnostics-configure-application-insights

Once you configure WAD to sink data to Application Insights, you can go into your Azure Subscription Portal, click on the instance of Application Insights that you have set up, then click on the Analytics button to get a very powerful ability to query all the incoming diagnostic data:

As the data comes in from WAD, it is routed by Application Insights to different 'tables', which are listed in tree form as child nodes under the parent node 'APPLICATION INSIGHTS'. You can then query these tables to see the data:

I had designated several performance counters to be captured by WAD and sent to Application Insights. Naturally, I assumed that I would find them in the table named performanceCounters (see above picture). However, I only found some of them showing up there, one of which was % Processor Time. I didn't know what happened to others and thought I had misconfigured WAD.

After some investigation, I found out that for legacy reasons, some performance counters are forced by Application Insights to the performanceCounters table. All other counters go to the customMetrics table.

As best as I could determine after speaking with various people, here is a list of counters that are routed by Application Insights to the performanceCounters table:

@"\Process(??APP_WIN32_PROC??)\% Processor Time",

@"\Memory\Available Bytes",

@"\ASP.NET Applications(??APP_W3SVC_PROC??)\Requests/Sec",

@"\.NET CLR Exceptions(??APP_CLR_PROC??)\# of Exceps Thrown / sec",

@"\ASP.NET Applications(??APP_W3SVC_PROC??)\Request Execution Time",

@"\Process(??APP_WIN32_PROC??)\Private Bytes",

@"\Process(??APP_WIN32_PROC??)\IO Data Bytes/sec",

@"\ASP.NET Applications(??APP_W3SVC_PROC??)\Requests In Application Queue",

@"\Processor(_Total)\% Processor Time"

All other counters are routed to the customMetrics table. There does not appear to be any configuration option to change where Application Insights routes the data, so I wanted to let those interested know that they'll need to query both the performanceCounters and customMetrics tables to retrieve performance counter values.

Controlling the Number of Outgoing Connections From HttpClient (.Net Core or Full Framework)

Tim Omta — Mon, 23 Oct 2017 13:18:05 GMT

Although there are references to this all over the web for full framework, I had trouble finding it for .Net Core, so I thought I'd note it for others that may be searching. On .Net Full framework, you can control the number of outgoing client connections by setting ServicePointManager.DefaultConnectionLimit to the desired number. It is 2 by default. On .Net Core, you can set it by changing HttpClientHandler.MaxConnectionsPerServer to your desired limit. The default is int.MaxValue -- much higher than the default for the full framework.

Add Azure Antimalware Extension to a VM Scale Set

Tim Omta — Wed, 08 Mar 2017 12:34:16 GMT

If you're running a Windows Server 2016 base image, ignore this post because we've built Windows Defender in. However, if you're running Server 2012R2, for example, and wish to apply the Microsoft Antimalware Extension to an existing VM Scale Set, here is a PowerShell script to do it:


# Add the Microsoft Antimalware Extension to an Azure VMSS
# Assumes you have already authenticated your PowerShell session
# with Login-AzureRMAccount and selected the correct subscription
# with Select-AzureRMSubscription

$rgname = 'myResourceGroupName'
$vmssname = 'myVMSSName'
$location = 'westcentralus' # change to the Azure Region of your VMSS
 
# retrieve the most recent major version number of the extension
$allVersions= (Get-AzureRmVMExtensionImage -Location $location -PublisherName "Microsoft.Azure.Security" -Type "IaaSAntimalware").Version
$latestVersion=$allVersions[-1]
$majorVersion=$latestVersion.Substring(0,$latestVersion.IndexOf('.', 2))

$VMSS = Get-AzureRmVmss -ResourceGroupName $rgname -VMScaleSetName $vmssname
Add-AzureRmVmssExtension -VirtualMachineScaleSet $VMSS -Name "IaaSAntimalware" -Publisher "Microsoft.Azure.Security" -Type "IaaSAntimalware" -TypeHandlerVersion $majorVersion
Update-AzureRmVmss -ResourceGroupName $rgname -Name $vmssname -VirtualMachineScaleSet $VMSS

A customer asked me how to do this. I didn't know, so I asked around, put it together, tested it, and thought I'd share. Special thanks to Guy Bowerman for help with the script and a quick review. References Azure VM Scale Sets Windows Defender Built into Server 2016

How to Trap Uncaught Exceptions and Avoid 5xx Errors in ASP.Net WebAPI

Tim Omta — Sat, 04 Mar 2017 10:42:59 GMT

I was recently writing a WebAPI 2.x project for a customer that involved file IO, decompression, and database IO. Though the things the API does are not complex, there are many things that can go wrong in any of those types of operations that can cause exceptions to be thrown (SQL parsing errors, IO errors, connection errors, etc). I took steps to catch and deal with the exceptions that I could anticipate, but there is always the potential for some to go unhandled, which results in a 500 class error from the WebAPI. 5xx errors are very generic “Internal Server” errors and difficult to debug in addition to being just plain ugly going back to the customer. WebAPI 1.x has Exception Filters, which cover many cases. You’ll also notice, as you go through the documentation, that it tells you how to write and register them. WebAPI 2.x has even more, global options that allow you to cover more cases, like exceptions thrown from controller constructors or during a response stream. The new options are 2 base classes, ExceptionLogger and ExceptionHandler, that you can inherit from and override methods on to gain control of how uncaught exceptions are handled. The goal of the first class is to allow you to insert a call to your logging framework to log the exception. The goal of the second class is to allow you to send the client an error message of your choosing rather than “500 Internal Server Error”. This looked great and was what I wanted to do, but though the doc explained the goals of global error handling, it didn’t explain how to properly write or register these options. Of course, like most projects, it had to be done yesterday and professionally because, well, that’s my job. In order to leverage these new exception handling options, I had to spend a good deal of time asking questions, doing web queries, downloading, and spelunking through sample code. I would like to save you that time by showing you how to implement register ExceptionLogger and ExceptionHandler. ExceptionLogger To implement ExceptionLogger, simply write your own class that inherits from it, as the docs show. At the time of this blog post, the docs do not show the proper override to write though. At a minimum, override the Log(ExceptionContext context) method. If you look at the source code for ExceptionLogger, you’ll see that the base implementation of LogAsync will wrap a call to your overridden Log method in a Task, so no need to override it unless you really need to do something special in the async case. To register ExceptionLogger, add this call to the Register(HttpConfiguration config) method of your WebApiConfig class: config.Services.Add(typeof(IExceptionLogger), new TraceExceptionLogger()); ExceptionHandler To implement ExceptionHandler, you again write your own class that inherits from it, as the docs show, but instead you must override the Handle(ExceptionHandlerContext context) method at a minimum. To register ExceptionHandler, add this call to the Register(HttpConfiguration config) method of your WebApiConfig class: config.Services.Replace(typeof(IExceptionHandler), new UncaughtExceptionHandler()); Here is a sample that I finally found that shows registration and implementation. It uses an additional package called ELMAH, but you don’t need that to implement what I’ve told you about here. The sample shows 2 very basic controllers that throw exceptions and how these are handled with an implementation of ExceptionLogger (ElmahExceptionLogger) and ExceptionHandler (GenericTextExceptionHandler). Registration of these classes is shown in WebApiConfig for the project. References https://docs.microsoft.com/en-us/aspnet/web-api/overview/error-handling/exception-handling https://docs.microsoft.com/en-us/aspnet/web-api/overview/error-handling/web-api-global-error-handling https://docs.microsoft.com/en-us/aspnet/web-api/overview/releases/whats-new-in-aspnet-web-api-21#global-error https://aspnet.codeplex.com/SourceControl/latest#Samples/WebApi/Elmah/ReadMe.txt

How to Remove the Server Header from WebListener

Tim Omta — Sat, 03 Dec 2016 16:04:45 GMT

Sometimes enterprises or those hosting web sites wish to remove the Server HTTP response header so they can make it more difficult for an attacker to determine what type of web server they are using (IIS, Apache, etc). Though it is doubtful as to how much protection this actually provides, it is still a common practice. ASP.Net Core includes a new web server called WebListener to offer you a light weight alternative to IIS. I was asked how to remove the Server header from WebListener by a customer. After some asking around and experimenting, the following solution turned out to be the best one at present:

public void Configure(IApplicationBuilder app)
{
    // Add code to the beginning of Configure to blank out
    // the header, which removes it
    app.Use((context, next) =>
    {
        context.Response.Headers.Add("Server", string.Empty);
        return next();
    });

    // continue with other pipeline configuration code...
}

Here is the response from a WebListener server before the above addition: Here is the response after the code was added: In the latter figure, you can see that the server responded without the Server header in the HTTP response.

How Do I Set Up A .Net Core WebListener With SSL?

Tim Omta — Fri, 04 Nov 2016 17:09:09 GMT

I've been doing a lot of research on Service Fabric and Windows Docker Containers lately. These are natural platforms for .Net Core, which added another learning item to my list: .Net Core. As a result, I set out to get a .Net Core WebListener web server up and running on Windows Server Core 2016. I chose Server Core because it emulates the kind of environment you'll be faced with when running on Docker or Service Fabric, which is essentially a bare bones, no UI environment. I'm not attempting to teach you Hyper-V, Docker, or how to get around in Server Core as they are big subjects by themselves. If you are afraid of the command line, you should not be here until you do some prerequisite learning.

Objective: Set up a .Net Core WebListener Server with SSL on Windows Server Core 2016

Create Your Server Core Host

Start by installing Server Core 2016 in a VM. You could use a Docker container too if you're so inclined.

Create Your SSL Certificate

Log onto the new Server Core machine. At the administrative command prompt, type

start powershell

Issue a PowerShell command to to create a self-signed cert (use your server's DNS name) New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName server0.contoso.com You'll need the cert thumbprint a little later. Mine was: C74235968895ECA9C9D50703D30CE17A7D3AEEFF

Register Your App With HTTP.SYS

Issue a PowerShell command to create a GUID New-guid You'll need this Guid to use as an AppId later. Mine was: 921328e8-5cdf-4b7f-a3cc-4e50147d1521 Switch back to your original administrative command prompt and issue these commands to create the Url ACLs (80 and 443)

netsh http add urlacl url=https://+:443/ user=Users
netsh http add urlacl url=https://+:80/ user=Users

While still in the same command prompt, issue a command to associate the cert thumb print and App Id (created above) with port 443

netsh http add sslcert ipport=0.0.0.0:443 certhash=C74235968895ECA9C9D50703D30CE17A7D3AEEFF appid={921328e8-5cdf-4b7f-a3cc-4e50147d1521}

Note: If you run the above from PowerShell rather than cmd, you may need to escape some characters to get it to work

Open the Firewall Web Ports

Switch to your PowerShell prompt Issue a PowerShell command to open 80 and 443 in the Windows Firewall. This will allow HTTP(S) traffic to WebListener.

New-NetFirewallRule -DisplayName 'HTTP(S) Inbound' -Profile @('Domain', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort @('80', '443')

I opened both ports because I wanted to test both clear HTTP and HTTPS. I also chose to restrict the ports from being open on the Public firewall profile by specifying Domain and Private.

Install .Net Core

Issue a PowerShell command to download .Net Core to the machine Invoke-WebRequest "https://go.microsoft.com/fwlink/?LinkID=827524" -OutFile .\DotNetCoreSetup.exe -UseBasicParsing In the same PowerShell prompt, install .Net Core using the following .\DotNetCoreSetup.exe /install /quiet /log .\log.txt .Net Core will silently (and very quickly) install. I chose silent because this is not a full GUI machine. When the install finishes, you should see several text log files in the directory you issued the above command from. Even though you're on Server Core, Notepad still works and you can use it to inspect the logs and satisfy yourself that you had a successful install. Once satisfied, I really recommend you restart the machine with this PowerShell command Restart-Computer You don't need to restart, but it is a pain if you don't let the machine pick up the environment changes.

Create a Web Application to Serve

Now we need an app. The easiest thing to do is to adapt the template web app. Step 1 - From your Server Core console, enter the following to create the template web app:


mkdir aspnetcoreapp
cd aspnetcoreapp
dotnet new web

Step 2 - Modify aspnetcoreapp.csproj file to include the WebListener package reference right under the Microsoft.AspNetCore package reference. Notepad is available on Server Core by typing 'notepad' at your command prompt. Your csproj should look like this:



  
    netcoreapp1.0

Step 3 - Modify Program.cs by replacing the WebHostBuilder initialization statement with this:


            var host = new WebHostBuilder()
                .UseWebListener()
		.UseUrls("https://+:80", "https://+:443")
                .UseStartup()
                .Build();

NOTE: Make sure the above URL strings "https://+:80" and "https://+:443" match the URL strings used in the ACLs you created with netsh earlier in 'Register Your App with HTTP.SYS'. If you don't, you may receive a 503 Service Unavailable from the web server. Step 4 - From your Server Core console, enter the following to restore the packages and run the server:


dotnet restore
dotnet run

Step 5 - Browse to your Server Core machine name from another machine on the network, using either HTTP or HTTPS. In my case, the URLs to browse to were the same as the DnsName for my cert and machine:


https://server0.contoso.com/
https://server0.contoso.com/

You should have a working SSL enabled WebListener server running on Windows Server Core 2016. Now go and brag about your .Net Core, bare metal, headless, command line skills. References ASP.Net Core Docs Windows Server 2016 WebListener repository WebListener ASP.Net Core Docs URL routing/registration/pattern matching behavior Service Fabric Documentation Containers for Windows

How Do I Open Ports with PowerShell?

Tim Omta — Fri, 04 Nov 2016 13:32:08 GMT

If you have a history with Windows, you're probably used to using netsh to open ports on the Windows Firewall. We have a whole slew of PowerShell cmdlets to administer the Windows Firewall now and I wondered how to use PowerShell to open ports. It took me longer than it should have to do it, so I thought I'd share. In the end it is simple, but there are a lot of cmdlets to wade through, which is where I got hung up. Here is what you'd do with netsh to open ports 80 and 443:


netsh advfirewall firewall add rule name="Open Port 80" dir=in action=allow protocol=TCP localport=80
netsh advfirewall firewall add rule name="Open Port 443" dir=in action=allow protocol=TCP localport=443

Here is how you'd open the same ports for only the Domain and Private profiles (not Public) with PowerShell:


New-NetFirewallRule -DisplayName 'HTTP(S) Inbound' -Profile @('Domain', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort @('80', '443')

References PowerShell cmdlets to manage the Windows firewall Netsh Examples

How to Use PowerShell DSC to Prepare a Data Drive on an Azure VM

Tim Omta — Sat, 23 Apr 2016 22:39:05 GMT

Introduction

I was asked how to automate formatting a data drive on an Azure VM. The drive had been provisioned already when the infrastructure was created via ARM script, however the disk was raw. I developed this solution with Azure PowerShell Desired State Configuration (DSC), Azure PowerShell 1.0 (newest release as of April 2016), and my Windows 10 client machine, meaning I have PowerShell 5 and can leverage the PowerShell Gallery. DSC was originally developed for Windows Server and has been extended into Azure.

Setup From Administrative PowerShell

The first step is to install the xStorage module from the PowerShell Gallery. PowerShell needs the NuGet provider to download module packages from the gallery. I issued the below commands from an administrative PowerShell session. Once you have completed this on your development client machine, you don't need to do it again.

PS C:\WINDOWS\system32> Install-Module xStorage

NuGet provider is required to continue

PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or

'C:\Users\tomta\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install andimport the NuGet provider now?

[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): y

Untrusted repository

You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from

'PSGallery'?

[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): y

Module Code

This part does not need to be done from an Administrative PowerShell. I just started ISE from a PowerShell prompt and used it to write my Desired State Configuration (DSC) script.

PS C:\temp> ise

In ISE, I wrote and saved the below to c:\temp\prepardisk.ps1

Configuration DataDisk

{

Import-DSCResource -ModuleName xStorage

Node localhost

{

xWaitforDisk Disk2

{

DiskNumber = 2

RetryIntervalSec = 60

Count = 60

}

xDisk FVolume

{

DiskNumber = 2

DriveLetter = 'F'

FSLabel = 'Data'

}

DataDisk -outputpath C:\DataDisk

Start-DscConfiguration -Path C:\DataDisk -Wait -Force -Verbose

The above will wait for disk volume 2 to show up connected to the Azure VM, which it should already be. It will then initialize, partition, format, and bring the volume up as F:.

Deployment

Next, we deploy the DSC configuration we want the VM to enact up to Azure, then tell he VM to update. You need an Azure storage account. The account is used to store the configuration so the VM can get it.

I performed these actions on a classic VM. If you're using ARM, the following steps are different and I need to write another post for it J. I'm also assuming that you have authenticated your PowerShell session by executing Add-AzureAccount, and using Select-Subscription to activate the correct subscription for your VM and storage account.

PS C:\temp> $key=Get-AzureStoragekey -StorageAccountName timsstorageaccount

PS C:\temp> $context=New-AzureStorageContext -StorageAccountName $key.StorageAccountName -StorageAccountKey $key.Primary

PS C:\temp> Publish-AzureVMDscConfiguration -ConfigurationPath C:\temp\preparedisk.ps1 -StorageContext $context

PS C:\temp> $vm = get-azurevm -ServiceName TimsCloudService -name dev1vm1

PS C:\temp> Set-AzureVMDscExtension -vm $vm -ConfigurationArchive "preparedisk.ps1.zip" -ConfigurationName "DataDisk" -StorageContext $context | Update-AzureVM

OperationDescription OperationId OperationStatus

-------------------- ----------- ---------------

Update-AzureVM 3f88ae8d-5cf0-a7c8-8e0f-f4c30324e96d Succeeded

Visually Check the Deployment

After a few minutes, I RDP'd into the Azure VM to be satisfied that all worked well. I opened Windows File Explorer and the disk showed up partitioned, formatted, and with the correct drive letter.