Helpful Table Of Problem Versions

• 10.0.14393.0 - Windows Server 2016 ships with a broken servicing stack which can't talk to WSUS. (15 months out of date)
• 6.3.9600.16xxx - (18xxx is current) means Windows Server 2012 R2 or Windows 8.1 without 2919355 applied (~3 years missing updates)
• 7.6.7600 - Windows Server 2008 R2 or Windows 7 without Service Pack 1 (5 years missing updates)
• 6.2.6001 - Windows Server 2008 Service Pack 1 (5 years missing updates)

• 10.0.14393.1670 - Win10 or Win 2016 with Sep 2017 CU (anything later than .187 is probably OK)
• 6.3.9600.18xxx - Win2012 R2 post-2919355
• 7.6.7601 - Win2008 R2 SP1 / Win7 SP1
• 6.2.6002 - Win2008 SP2

Back In The Day, Build Numbers Were Even More Useful

What can you do with this information?

What should you do?

• Start, Run, WinVer on a suspect PC (if it doesn't say Service Pack X, problem)
• PS:   get-adcomputer -Filter '(OperatingSystemVersion -like "*7600*") -or (OperatingSystemV ersion -like "*6001*")' -Properties OperatingSystemVersion,OperatingSystemServicePack | export-csv NoServicePack.csv           #  (a blank NoServicePack.csv = good)
• Or    wmic /node:servername os get version     - if WMI (RPC) is enabled to the target (in which case, extra bonus security points lost unless you're using a PAW or management host - you should be firewalling!)
• Or use WSUS: Turn on the Version column in the All Computers view in the WSUS console, then Group By (or just Sort by) Version and look at the build numbers reported. (Don't forget to filter by Any)

And

And And: an afterthought for 2012 R2

For all supported x64-based versions of Windows 8.1 and Windows Server 2012 R2

For all supported x64-based versions of Windows 8.1 and Windows Server 2012 R2

I wanted to add text-based conditional formatting to an Excel sheet I was creating from PowerShell – so I could colour one of the columns automatically depending on the values.

I used the technique any self-respecting dabbler would: I recorded a macro in Excel VBA and then tried to convert it over.

But no matter what I tried, I couldn’t get the !&$^$^@ FormatConditions.Add function to work.

After debugging the idiot-level mistakes (null variables; debugging without parameters :| ) out of the script, I was left with:

Type mismatch. (Exception from HRESULT: 0x80020005 (DISP_E_TYPEMISMATCH))
At C:\Users\tristank\Desktop\excelfmt.ps1:201 char:1
+ $newthing=$ActionColumn.FormatConditions.Add($xlTextString, $cond, $x ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], COMException
    + FullyQualifiedErrorId : System.Runtime.InteropServices.COMException

Soo… not enough arguments, then?

The VBA reference which comes up first in Bing (and Google for that matter) wasn’t helpful – 4 arguments and I’d tried every combination of arguments I could conceive. The count seemed fine. But no FormatConditions object…

But debugging from the command line (thanks ISE), I tried punching in $ActionColumn.FormatConditions.Add( , and was stunned to see 7 arguments in the tooltip.

I added $null to all of them (natch) – and it worked! (Or at least I had a rule with the right conditions and formatting, but the condition was in the wrong spot).

So, armed with this new knowledge, I found the Excel Interop Object Reference for FormatConditions.Add … and there ya go. 7 arguments.

Object Add( XlFormatConditionType Type, Object Operator, Object Formula1, Object Formula2, Object String, Object TextOperator, Object DateOperator, Object ScopeType )

As I’d already been experimenting by the time I found the documentation, it seems like several bits might be interchangeable. I don’t know what I’m doing; I can’t warrant that this will work under any conditions except my own.

But to save someone else the time, here’s mine:

$ActionColumn = $worksheet.Range("G:G")
$xlTextString = [Microsoft.Office.Interop.Excel.XlFormatConditionType]::xlTextString
$xlContains = [Microsoft.Office.Interop.Excel.XlContainsOperator]::xlContains
$cond = "ACTION"
$newthing=$ActionColumn.FormatConditions.Add($xlTextString, "", $xlContains , $cond, $cond, 0, 0) # hacky dodgy hacky hack / works

$fcs=$ActionColumn.FormatConditions.Count
$ActionColumn.FormatConditions.item($fcs).SetFirstPriority()
$ActionColumn.FormatConditions[$fcs].Font.ThemeColor = [Microsoft.Office.Interop.Excel.XlThemeColor]::xlThemeColorDark1
$ActionColumn.FormatConditions[$fcs].Font.TintAndShade = 0
$ActionColumn.FormatConditions[$fcs].Interior.Color = 255
$ActionColumn.FormatConditions[$fcs].Interior.TintAndShade = 0
$ActionColumn.FormatConditions[$fcs].StopIfTrue = $false;
   

And yes, I was naughty and used two indexing operators (during my “long script debugging” phase, when I was troubleshooting an uninitialized object and didn’t know it).

Sigh. And hope that helps!

I can never find the instructions I used to fix it, so I thought I’d post them here.

Note: For Surface Pro devices only.

Hold Power and Volume Up for 15 seconds. The screen may do things (turn off, turn on. weird clown images appearing) during this time, but just keep pressing and holding.

After 15 seconds, release the buttons, and then press the Power button to boot the Surface again.

And check to see how things are…

An 8 bit rendition of a scary, scary ghost. It scared me.

If you’ve seen it, you’re not alone. It’s a thing.

I hope - one day - to capture a screenshot of this apparition.

The Rules appear to be that you can never be ready for a screenshot when RDP Ghost appears.

Update 22/04/17: I WAS READY! Kinda. And caught enough that I can understand the cause and the effect.

The good news is, you’re probably not hacked by a supernatural force. Probably. It’s just Windows trying to draw the sign-in guy before it’s ready!

Krebs’s Immutable Truths About Data Breaches

“There are some fairly simple, immutable truths that each of us should keep in mind, truths that apply equally to political parties, organizations and corporations alike:

-If you connect it to the Internet, someone will try to hack it.

-If what you put on the Internet has value, someone will invest time and effort to steal it.

-Even if what is stolen does not have immediate value to the thief, he can easily find buyers for it.

-The price he secures for it will almost certainly be a tiny slice of its true worth to the victim.

-Organizations and individuals unwilling to spend a small fraction of what those assets are worth to secure them against cybercrooks can expect to eventually be relieved of said assets.”

D:\WebContent> dir /s *.txt

• There is no reason for files like readme.txt to exist within your production website
  • Okay, there’s one reason and that’s when you’re providing one you know about, and have vetted, for download.
    • I mean, obviously if the site is there to provide readme.txt s for apps people are downloading, great! But if it's the readme for some developer library which has been included wholesale, bad pussycat.
• There is no reason for files like web.config.bak to exist within your production website.
  • Luckily, .bak files aren't servable with the default StaticFileHandler behaviour. But that doesn't mean an app (or * scriptmap...) can't be convinced to hand you one...
• If you have web.config.bak.txt files, you’re asking for trouble.
  • Change your operational process. Don’t risk leaking usernames and passwords this way.

The Core Rationale

• It’s safe to back up to F:\Backups\CoolNewSite\2016-11-13 because it’s outside the servable website
• It’s not cool to back up to F:\Websites\CoolNewSite\2016-11-13 because that’s part of the website.

How Do I Know If I’m Crufty?

• .\Get-CruftyWebFiles.ps1   # scans all web content folders linked from Sites, and outputs to .\crufty.csv
• .\Get-CruftyWebFiles.ps1 -WebSiteName "Default Web Site"     # limits to just the one website.
• .\Get-CruftyWebFiles.ps1 -DomainName "YOURDOMAIN"    # checks for that text string used in txt / xml files as well

• You know the base product is in Extended Support until 2020, then it’s going away. (sniff!)
• You understand that Malware Scanning and Network Inspection System are already frozen at their last update level.
• You know URL Categorization (Filtering) got turned off already so any rules using it might fail-open (or fail-closed)…

• You’ve also been through your rule set, and tested that everything’s Least Privilege-compliant,
  • i.e. No broad “everyone can access anything/TMG/anywhere with any protocol” rules or anything like that.
    • No really, if you can connect to TMG via SMB, that’s usually not a good sign… You’re at least using Windows Update for patches, though, right?
• Maybe you’ve performed an ISAINFO (and/or TMGBPA) export of your rule set so that you can ease the process of recreating them on the next egress device you pick? :)

• Install the latest Windows Updates
• Install the latest TMG Rollup Hotfix (SP2 UR5, potentially + .650 or later)
• (Install any updates for any other software on the box)

• Firewalling
• De-Adminning
• Attack Surface Reduction
• AV exclusions

• Check Tracing isn't enabled
• Disable/Relax Flood Prevention

Proactively Protect The Box

Install the latest Windows Updates

• If you’re not installing Windows updates, um, I don’t know what to tell you?

• When I check the update state of a box, I do so by running MBSACLI (the command-line version of MBSA) using the current WindowsUpdate CAB if the box doesn’t have Internet connectivity.

mbsacli /xmlout /nvc /nd /wi /catalog .\wsusscn2.cab /unicode > %computername%-MBSA.xml

• • I actively avoid using the default customer WSUS catalog, because it’s completely possible to be 100% compliant with the WSUS approval policy and have unapproved updates missing from five years ago, which were skipped for a good reason, but then that decision was never revisited.
• It is uncommon in my experience to find that servers are up to date. For a security appliance at the edge of the network, used as an ingress or egress point by thousands of clients, this is suboptimal.

Windows Server 2008 R2 Service Pack 1 is needed for Security Updates

• So the first thing I’d check is WinVer.
• If WinVer says you’re on Windows 2008 R2 version 7600 and doesn’t mention a Service Pack, you need to get to 7601 (Service Pack 1) pronto, and then start applying all the updates which have required SP1 - say, the last 4-5 years’ worth, which includes many Critical updates.
• Windows 2008 should be at SP2. If it’s not at SP2, same thing applies as above.

If You Found You Had Something Missing: Why Not Just Use Windows Update?

• If you find they’re missing updates because { ¯\_(ツ)_/¯ }, my standard remediation suggestion is: just point them at public WindowsUpdate and specify your schedule. Let them pop out through a proxy, or go direct if they’re edge devices.

Install the latest TMG Rollup Hotfix

• The latest rollup version I’m aware of is TMG Service Pack 2 with Update Rollup 5. If Help/About in the TMG MMC shows you a version earlier than 7.0.9193.644, well – that update was from 2014.
• There’s one post-rollup hotfix I’ve seen (which is for SNI websites with HTTPS inspection enabled, but it provides a version bump to .650 for many core components too) which gets us to April 2015: https://support.microsoft.com/en-us/kb/3058679 .

Operating System Protection

Lifecycle and post-Lifecycle Firewalling

• I don’t mind restating the obvious, so I will: You should have migrated away from TMG before the end of 2019. Please!
  • That’s still 3 years from now to plan and execute your migration
  • So if you haven’t already started, please add it to your “To Do: 2017” list now.
• If you do still have some TMG kicking around at that point, consider hardening the TMG Firewall policies (including the System policies) to limit all nonessential connectivity to the TMG hosts by any other computer.
  • In fact, think about doing that anyway, particularly if you actually had work items pending from the “Install Windows Updates” item above. Because that's an attack surface exposure compounded with known vulnerabilities. That's a poor combination for a security device.

De-Adminning

• Just check the membership of any groups who have Admin permission to the box.
• Then eliminate any local admins except one (if you don’t fully de-admin boxes), and remove any Domain groups you can.

Basic Attack Surface Reduction

• If you have looked at them in the last 6 months, you can be excused from this item.
• If not, check to see what the file dates of the EXEs are. If they’re over 3 years old, they’re probably a liability and almost certainly aren’t being updated, and simply represent an increased attack surface, so consider removing them.

Antivirus

TMG Health

Tracing?

• Run RESMON for a short while, look at the Disk IO area, and sort by Bytes Total/sec. Note any files which have lots of IO over a 2 minute period.
  • (The idea is to try to minimize IO where possible)
• If activity to ISALOG.BIN is chewing through a megabyte or more per second, TMG may be tracing something
  • or still tracing something - this has been seen when TMGBPA is used to run a diagnostic trace but for whatever reason it doesn’t terminate cleanly.
  • It might also indicate a diagnostic logging session is in progress (just check the console under Troubleshooting –> Diagnostic logging and hit Disable if it isn’t already disabled).

Flood Mitigation

• Try a 10X increase in the numbers, particularly for HTTP and TCP connections, and see how you go.
  • If it stops the constant alerting about “infected clients”, and you’ve got burnout from chasing them down only to find it was Bruce in Marketing opening eighteen instances of FireFox to their brand new multi-pronged CDN-driven site manually, it might be a welcome change, and reduce grumbling (nothing like a paused connection to cause a user to get grumpy about “the )(@$& Proxy”)…

Hi Tristan,

Do you have by any chance a guide on how to set up IIS for kerberos auth? I'm helping my customer and I'm a beginner with IIS.

It is a farm of 6 IIS servers, they will be using a service acct.

DNS is configured to do the following resolution:

Websvr -> CNAME -> IP

So for instance the web site is webapp.example.net and points to a CName. The CName obviously is an fqdn (app-prod-vip.example.net) that points to an IP.

The IP points to the VIP of a load balancer that ultimately connects to the IIS server farm.

When setting the SPN do we use the websvr or the CName?

Also, does it matter the browser I'm using on the client for kerberos auth (such as chrome)

Anything special on the web server, besides configuring Windows authentication?

Thank you!

Here’s what I replied with:

Hola!

Couple of moving parts there – it (a different name, i.e. the load balancer name) won’t work with the default configuration.

You’ll need to ensure that the SPN for the CNAME is only assigned to the service account running the App Pool. If it’s on more than one account, it’s broken.

A DA needs to run:

SetSPN -S http/cname-of-app.fqdn.com DOMAIN\AppPoolAccountName

Where DOMAIN\AppPoolAccountName is the service account you set up for the application.

And that should get kerb where it needs to be from an SPN perspective. If other SPNs have been tried already, they need to be removed (and SetSPN -S should tell you that).

(Once you’ve established an SPN for the account, the Delegation tab should appear for it in ADUC. This allows you to configure constraints or delegation, which you might not be doing, so we’ll cover that last.)

Next, you need to ensure the App Pool Account is set to DOMAIN\AppPoolAccountName (i.e. the same “custom” domain account) on all the boxes. (ApplicationPoolIdentity or NetworkService or LocalSystem or anything other than a Domain account won’t work for load-balanced Kerberos authentication.)

Then, you need to either

• disable Kernel-mode authentication, or
• set useAppPoolCredentials=true

on them all.

There’s a tickbox for K-mode auth under Windows Authentication in IIS; or useAppPoolCredentials goes (I think) in web.config so might be preferable. What either of these does is to move from using the box identity (machine account) to validate tickets, to using the App Pool Account to validate tickets. This is required for a farm scenario, but for a single-box scenario, it’s not necessary (only SPN registration).

Once that’s done, Kerb should work to the websites, which can be validated with a network trace, or by looking at logs. (let’s throw in a reboot after k-mode auth is toggled off for good measure) (Picking Kerb in logs – short version: single 401 www-auth:negotiate/request with long ticket/200 is Kerb, 401/401/200 is NTLM).

I’d always test with IE, I *think* if IE works then Chrome has a good chance. If it doesn’t, no chance

Always test from a remote box (avoids reflection protection), and use klist purge (and a closed browser) to reset between tests.

If Kerb works to the site, you can then configure the App Pool Account in ADUC for constrained delegation to the next hop in the same way. Hit Add, browse for the process identity it’s connecting to (i.e. often the service account if the process is running as a domain identity, not the box name, but if not, the box name) and then pick the right SPN from the list.

HTH!