Online memory of an Active Directory PFE

A Domain Controller is not a Domain Computer

Willem Kasdorp — Mon, 10 Dec 2018 14:37:28 GMT

Today I spent half a day troubleshooting an issue with Authentication Silos that I finally tracked down to an unexpected issue with a Group Policy. To give you a little background information: Authentication Silos are used to limit to which computers a user can log on. This is especially useful in Tier-0 configurations. To enable Authentication Policies you must enable Kerberos Armouring on both clients and KDCs, which is needed to support Silo claims. These switches are set through Group Policy. So what happened was that a random team member decided to do some Group Policy hardening using security filtering. He wanted to make sure that users would not apply the policy set for Authentication Silos, so he removed the permission Authenticated Users:Apply and replaced it by domain computers:Apply. The next morning the Silo was broken. Nobody could get in. Because there are quite a number of moving parts to Authentication Silos it took us a while to get down to the problem: no claims were generated, because the Group Policy was not applying to Domain Controllers. Which is pretty weird, because surely a Domain Controller is member of Domain Computers? Guess again: A DC is member of group Domain Controllers but not of Domain Computers! A little digging revealed the reason for this. These groups are designed as primary groups, and user or computer can only be member of one primary group. A normal computer has Domain Computers as its primary group. Once it gets promoted to a DC, the primary group changes to Domain Controllers, meaning that it leaves membership of Domain Computers... Most (all?) of the guidance on Group Policy filtering gets this wrong, including the well-read blog Deploying Group Policy Security Update MS16-072 \ KB3163622 om remediating MS16-072:

If adding "Authenticated Users" with just "Read" permissions is not an option in your environment, then you will need to add the Domain Computers group with "Read" Permissions

So, the bottom line: if you use the group Domain Computers for any form of security filtering, be aware that you need to add DCs explicitly to the ACL using group Domain Controllers.

What's new in Active Directory 2019? Nothing.

Willem Kasdorp — Sun, 02 Dec 2018 15:44:42 GMT

OK, so there is not precisely "nothing" new in AD 2019, but as a management summary it will do. Before you read on I would like to make it perfectly clear that:

This information is not official in any way.
All information here is based on public information.

I had a look in our documentation for new functionality in Active Directory and I found ... nothing. So I dug around a bit and started with the updates to the schema because new Active Directory features generally require a schema update. And yes, there is indeed an update. There is just one new file called sch87.ldf which adds just one new attribute called msDS-preferredDataLocation and raises the schema version to 88. The new attribute is associated with users, groups and contacts. It is documented on MSDN which indeed says that it's a post-2016 update. What does it do? My best guess is that it is related to Azure Active Directory which also has this attribute. That's all I could find for now. Update 12/2/2018: see the comments for some useful discussion. Another good question would be: are there new forest or domain functional levels? And the answer is, for the first time: no. The highest functional level offered by dcpromo and other GUI tooling is "Windows Server 2016". That's interesting. In all earlier versions of Active Directory you could use the Domain Functional to make sure you could not install DCs with earlier operating systems. Because there is no functional level for Windows Server 2019 you cannot enforce using Windows Server 2019 DCs only. At best you could enforce a mix of 2016 and 2019. Other than that I am aware of some bug fixes and internal optimizations, but that's about it. I guess that the good news is that Active Directory backwards compatibility with Windows Server 2016 will be very good. Any application or device that works with Windows Server 2016 should have no problem with Windows Server 2019. As usual, "working" and "supported" are very different concepts, so keep that in mind. In particular Exchange and Lync (sorry, Skype for Business) are famous for being very strict with supporting new Active Directory versions. So, summarizing, what new stuff do we have for Active Directory 2019 compared to Active Directory 2016?

one new attribute with an as-yet unknown function.
no new functional levels, which is a first.
Backwards compatibility should be better than ever.

Almost nothing 😃 Update 12/2/2018: one of the "internal optimizations" that I was aware of is an improvement in handling the Version Store, which is a memory buffer needed to handle database transactions. If you run out of space here, bad things happen such as the DC stopping to respond. My colleague Ryan Ries has now written a really (really) extensive blog on this improvement: Deep Dive: Active Directory ESE Version Store Changes in Server 2019. Considering that he wrote the code himself he deserves a little indulgence here.

Quickly find potential Kerberoast victims

Willem Kasdorp — Wed, 25 Apr 2018 00:32:12 GMT

Lately I have been talking a lot about the Kerberoast exploit with my customers. Before I dive in let me start off by giving you the two great resources that I used myself to get up to speed:

What is this aptly named Kerberoast exploit? It is a way to obtain the password of a service account, abusing the Kerberos protocol in a very elegant way. For this (conceptual) discussion I am going to assume that you know your way around Kerberos. If not (quite), read up at: Kerberos for the Busy Admin. The central idea is that if you can get hold of a ticket generated for a particular service, you can use this to bruteforce the password: the correct password is the one that generates the hash that in turn is used to reconstruct the ticket. Same ticket as the original one? You just found the password. The brilliant part of the idea is that it turns out to be very simple to get such a ticket. No need to put wireshark on the network to wait for something interesting. Instead, you just ask for the ticket! Well, there is a little more to it, but not much. Any Kerberos service must have a Service Principle Name (SPN), such as HOST/s1.sol.local. All computer accounts have SPNs, but are completely unsuitable for bruteforcing because they have 128-byte random passwords. The same holds for Managed Service Accounts. On the other hand, normal user accounts do not have SPNs and cannot be used to represent a Kerberos service. The exceptions are user accounts that do have SPNs and are used as service accounts representing a network service such as SQL, a web service, ADFS, etc. And you know user accounts: their passwords are certainly not 128-byte random binary strings! If you know of a user account with an SPN, you can simply ask for a Kerberos Service Ticket for this account using nothing more than Domain User credentials. That's right, no admin permissions anywhere. Any normal domain user can do this which is another aspect that makes Kerberoasting so interesting. Once you have the ticket, you can start guessing the password that generates the hash, which generates the ticket. And by the way, nothing stops you from asking for tickets for different services at the same time, and try to bruteforce them in parallel. Note that this bruteforcing is invisible to you as an admin. There is no network traffic involved anymore once the ticket is issued, no logon attempts to DCs, nothing like that. Instead, all the work is local to the machine, executed at full speed of the CPU, for as long as it takes. The ticket is only invalidated when the account behind it changes its password. A technical complication here is that AD accounts can have multiple hashes. Without going into details, the two most relevant hashes nowadays are the RC4 and AES hashes. And you guessed it, only the tickets based on RC4 hashes are easy to bruteforce. Unfortunately, this is the greater majority in most networks. The point of my post is to warn you for the suddenly popular Kerberoast approach. The most interesting targets will be highly privileged service accounts such as the ones that are member of Domain Admins. But there are also others cases, such as accounts giving access to important data, or those that are configured for Kerberos delegation. I wanted to show you a simple, easy to understand script that shows you which privileged accounts have SPNs. I put it up on GitHub: Find-PrivUsersWithSPN.ps1. Run it with no arguments and it will analyze the current domain and show the users having an SPN, with a flag telling you if it belongs to a privileged group. So in this case I have one user that I need to investigate right away: srv-agpm is apparently privileged. It turns out to be a member of Backup Operators. Not great, but no disaster either. Again, for the full details, see the links at the beginning of this post. What I'm writing here is just derived work that does not retrieve all relevant details. It just reveals the accounts that you should focus on first. What about remediation? You have several options.

Make sure that the password is really long, long enough to make bruteforcing impractical. Say, 25 characters or more.
Force the service account to use AES.
Move to using Managed Service Accounts.
Stop using highly privileged service accounts...

Logging on to Azure for your everyday job

Willem Kasdorp — Sun, 11 Feb 2018 21:43:25 GMT

Sometimes life is about the little things, and one little thing that has been bothering me is logging on to Azure RM in Powershell using Add-AzureRMAccount. Every time you start Powershell, you need to log on again and that gets tired quickly, especially with accounts having mandatory 2FA. It gets even more complicated if you have multiple accounts to manage, for instance, one for testing and another for production. To top it off, you can start over when it turns out that your context has expired, which you will only discover after you actually executed some AzureRM commands. The standard trick to make this easier is to save your Azure RM context with account and subscription information to a file (Save-AzureRMContext), and to import this file whenever you need (Import-AzureRMContext). But we can do a little bit better than that. Here is my solution to the problem.

Use a PowerShell profile to define a function doing the work. A profile gets loaded whenever you start PowerShell. There are multiple profiles, but the one I want is for CurrentUser - Allhosts.
The function will load the AzureRM context from a file. If there is no such file, it should prompt me to log on.
After logging on, the context should be tested for validity because the token may have expired.
If the token is expired, prompt for logon again.
If needed, save the new context to a file.

So here is my take on it. Note the specific naming convention that I use for functions defined in a profile.

function profile_logon_azure ([string] $parentfolder, [string]$accountname)
{
    $validlogon = $false
    $contextfile = Join-Path $parentfolder "$accountname.json"
    if (-not (Test-Path $contextfile))
    {
        Write-Host "No existing Azure Context file in '$parentfolder', please log on now for account '$accountname'." -ForegroundColor Yellow
    } else {
        $context = Import-AzureRmContext $contextfile -ErrorAction stop
        #
        # check for token expiration by executing an Azure RM command that should always succeed.
        #
        Write-Host "Imported AzureRM context for account '$accountname', now checking for validity of the token." -ForegroundColor Yellow
        $validlogon = (Get-AzureRmSubscription -SubscriptionName $context.Context.Subscription.Name -ErrorAction SilentlyContinue) -ne $null
        if ($validlogon) {
            Write-Host "Imported AzureRM context '$contextfile', current subscription is: $($context.Context.Subscription.Name)" -ForegroundColor Yellow
        } else {
            Write-Host "Logon for account '$accountname' has expired, please log on again." -ForegroundColor Yellow
        }
    }
    if (-not $validlogon)
    {
        $account = $null
        $account = Add-AzureRmAccount
        if ($account) 
        {
            Save-AzureRmContext -Path $contextfile -Force
            Write-Host "logged on successfully, context saved to $contextfile." -ForegroundColor Yellow
        } else {
            Write-Host "log on to AzureRM for account '$accountname' failed, please retry." -ForegroundColor Yellow
        }
    }
}

To make this work, add this function to the powershell profile: from the Powershell ISE, type ise $profile.CurrentUserAllHosts to edit the profile and copy/paste the function definition. Suppose I have two Azure RM accounts that I want to use here, called 'foo' and 'bar'. For that I would add the following function definitions to the profile:

#
# specific azure logons. Context file is deliberately in a non-synced folder for security reasons.
#
function azure-foo { profile_logon_azure -parentfolder $env:LOCALAPPDATA -accountname "foo" }
function azure-bar { profile_logon_azure -parentfolder $env:LOCALAPPDATA -accountname "bar" }

To log on to 'foo', I'd simply execute azure-foo. If this is a first logon, I get the usual AzureRM logon dialog and the resulting context gets saved. The next time, the existing file is loaded and the context tested for validity. From that point on you can switch between accounts whenever you need. Try it, it just might make your life a little bit easier.

Azure Batch for the IT Pro - Part 2

Willem Kasdorp — Fri, 12 Jan 2018 23:56:57 GMT

This is the second and final part of a blog series with a walkthrough for Azure Batch. The first part is here:

Azure Batch for the IT Pro - Part 1

In the first part I showed you how to create an Azure Batch Account, the corresponding Storage Account, a test application based on Powershell, and a pool of VMs to run the application. In this second part we will actually do the work of calculating Mersenne Prime Numbers. At this point, you should have a working Azure Batch account with a pool of VMs and a provisioned application. The first step is to retrieve the configuration.

Retrieve the Azure Batch configuration

Note that the code shown in this post is working, but was taken from Create-BatchJobsAndTasks.ps1 and simplified: no logging or error checking. Refer to the Azure-Batch GitHub project for the full scripts.

The code below is straight from the script, stripped from comments and additions. You can either execute Create-BatchJobsAndTasks.ps1 after downloading, or run the snippets below one-by-one. First, the configuration parameters. These should match the ones from the creation script discussed in the first post, Create-BatchAccountMersenne.ps1. The indexes here refer to Mersenne numbers; 20 is not high. Anything larger than 30 will take a measurable amount of time, indexes above 40 will take hours or days (and generate numbers with millions of digits).

$firstindex = 1
$lastindex = 20 # current maximum index.
$jobnamePrefix = "mersenne-job"
$tasknameprefix = "task"

$Applicationname = "Mersenne"
$ResourceGroupName = "rg-batch-walkthrough"
$BatchAccountNamePrefix = "walkthrough"
$ShareName ="mersenneshare"

The following helper function is needed to reconstruct the actual names of the Azure Batch and Storage account, as explained in the first post.

function Get-LowerCaseUniqueID ([string]$id, $length=8)
{
    $hashArray = (New-Object System.Security.Cryptography.SHA512Managed).ComputeHash($id.ToCharArray())
    -join ($hashArray[1..$length] | ForEach-Object { [char]($_ % 26 + [byte][char]'a') })
}

Read the Resource Group, reconstruct the Azure Batch account name from the Resource Group ID, and read the access keys of the Batch Account.

$ResourceGroup = Get-AzureRmResourceGroup -Name $ResourceGroupName -ErrorAction stop
$BatchAccountName = $BatchAccountNamePrefix + (Get-LowerCaseUniqueID -id $ResourceGroup.ResourceId)
$batchaccount = Get-AzureRmBatchAccount | Where-Object { $_.AccountName -eq $batchaccountName } -ErrorAction stop
$batchkeys = $batchaccount | Get-AzureRmBatchAccountKeys

We need the storage account to reconstruct the parameters ($Storagekey, $uncPath, $shareAccount) needed to access its SMB share. These parameters must be passed to the application what we will run on the nodes.

$StorageAccountName = "sa$($BatchAccountName)"
$StorageAccount = Get-AzureRmStorageAccount -ResourceGroupName $ResourceGroupName -Name $StorageAccountName -ErrorAction stop
$StorageKey = ($StorageAccount | Get-AzureRmStorageAccountKey)[0].Value
$Share = Get-AzureStorageShare -Name $ShareName -Context $StorageAccount.Context
$uncPath = $Share.Uri -replace 'https://','\\' -replace '/','\'
$shareAccount = "AZURE\$($StorageAccount.StorageAccountName)"

The final step before we can do real work is to retrieve the configuration of the VM pool. We will submit the job and tasks to this pool.

$pool = Get-AzureBatchPool -BatchContext $batchkeys | Where-Object { $_.State -eq "Active" }
$PoolInformation = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSPoolInformation" 
$PoolInformation.PoolId = $pool.Id

Create Batch Job and Tasks

A job is basically a container for tasks. It has a name, can be disabled or enabled, and is dedicated to a pool in an Azure Batch account. The name contains a timestamp, making it more convenient to track what happened and when. If job creation fails for some reason, we require a hard stop.

$jobnamePostfix = (Get-Date -Format s) -replace ':', ''
$jobname = "$jobnamePrefix-$jobnamePostfix"
New-AzureBatchJob -BatchContext $batchkeys -Id $jobname -PoolInformation $PoolInformation -ErrorAction Stop

Once the job exists, tasks can be submitted. The tricky part is to construct the commandline argument. We need to do a little pre-work: creating a unique name using a random generator, constraining the task to have 3 retries on failure, and retrieving the application definition to run.

$taskPostfix = Get-Random -Minimum 0 -Maximum 1000000
$constraints = New-Object Microsoft.Azure.Commands.Batch.Models.PSTaskConstraints -ArgumentList @($null,$null,3)
$batchapp = Get-AzureRmBatchApplication -AccountName $batchaccountName -ResourceGroupName $batchaccount.ResourceGroupName -ApplicationId $Applicationname -ErrorAction stop
$version = $batchapp.DefaultVersion
$appref = New-Object Microsoft.Azure.Commands.Batch.Models.PSApplicationPackageReference
$appref.ApplicationId = $batchapp.id
$appref.Version = $version

Finally, tasks are submitted to the VM Pool. For each Mersenne prime to be calculated, a new task is created. Each task gets put into a queue. The queue is used to submit tasks to Pool nodes (VMs). A next task gets scheduled only after successful completion, or definite failure of the previous task. Non-definite task failures are simply retried. The important bits: $ps1file is a hardcoded reference to the glue script generate_decimal_mersenne_and_upload.ps1. This script accepts the commandline arguments (an index to a Mersenne prime, and parameters for the SMB share). The taskname must be unique, and is constructed from a prefix, the index, and a random postfix. The commandlet New-AzureBatchTask submits the task and does not wait for its completion.

$firstindex..$lastindex  | ForEach-Object {
    $ps1file =  "%AZ_BATCH_APP_PACKAGE_MERSENNE#$($version)%\generate_decimal_mersenne_and_upload.ps1"
    $taskCMD = "cmd /c `"powershell -executionpolicy bypass -File $ps1file -index $_ -uncpath $uncPath -account $shareAccount -sakey $StorageKey`""
    $taskName = "$tasknameprefix-$_-$taskPostfix"
    New-AzureBatchTask -JobId $jobname -BatchContext $batchkeys -CommandLine $taskCMD -Id $taskname -Constraints $constraints -ApplicationPackageReferences $appref
}

If you would dump $taskCMD, it might look like the following (credentials are randomized):

cmd /c "powershell -executionpolicy bypass -File %AZ_BATCH_APP_PACKAGE_MERSENNE#1.0%\generate_decimal_mersenne_and_upload.ps1 -index 20 -uncpath \\sawalkthroughpkvydrcf.file.core.windows.net\mersenneshare -account AZURE\sawalkthroughpkvydrcf -sakey RCjwDatDd2TXgugAA74cfVUBqWROYvEYiEls0dKtXdD5zff4uOBW+SkHwonIG8iNJDG1kTf9anmKmrgbjBqWAZ=="

At this point, tasks should be running. To monitor for completion you could do something like the following. It gets the current state of all tasks in the job, summarises them, and if there are any unfinished tasks, sleep for three seconds. When done, terminate the job because there is no more work to do. This should not take long if you used the script defaults because it generates only the first 20 Mersenne primes, all of which are not very large.

do {
    $stats = Get-AzureBatchTask -BatchContext $batchkeys -JobId $jobname | Group-Object -NoElement state
    $stats | Format-Table
    $ready = ($stats.Values -notcontains "Active") -and ($stats.Values -notcontains "Running")
    if (-not $ready) { Start-Sleep -Seconds 3 }
} until ($ready)
Stop-AzureBatchJob -id $jobname -BatchContext $batchkeys

Inspect the results

With all the work done it's time to look at the Azure Portal and to retrieve the results. Let's start with the jobs and tasks. Go to the Resource group rg-batch-walkthrough, select the Batch Account, then Jobs (there should be just one initially), select this job, then select Tasks. The portal view should list the completed tasks, as follows. Pick any task you like, open it, select Files on Node. This will show you the list of files in the Working Directory of this particular task. This will always contain the stdout and stderr streams, which is very handy for debugging. It also shows the output file from the Mersenne calculation. You can download this if you like. But since we made a point of saving the output to an SMB share, there are also other ways to get at the data. For instance, if you have access to the storage account using the SMB protocol (445/tcp) you can access it directly, for instance from an Azure VM. Accessing it from your home or company network is very likely to fail because few ISPs allow 445 to/from the Internet. Alternatively, use Azure Storage Explorer, a tool to manage Azure Storage Accounts. You really should have a look at this if you are not familiar with it. Suggested exercise:

generate a couple of really large primes, such as the indices 42 or 43. If you are feeling brave and are prepared to wait a couple of days, try the largest one: 49.

Azure Batch for the IT Pro - Part 1

Willem Kasdorp — Fri, 12 Jan 2018 23:56:57 GMT

I spent some time on working with Azure Batch for a customer, and what struck me that it was not so easy for an IT Pro to create a meaningful testing setup. The stumbling point is that you need to have an application doing meaningful work. So what is Azure Batch? It is the PaaS version of High Performance Computing (HPC). Azure gives you the infrastructure, you give it the application and define the tasks and jobs telling Azure what to do. Azure Batch is suited to intrinsically parallel workloads, such as scientific model calculations, video rendering, data reduction, etc. My goal here is to set up an example that means something to the IT Pro who, like me, is familiar with Powershell but does not write .NET code in Visual Studio every day. The project is hosted on my Azure-Batch repo on Github.

Overview Azure Batch walkthrough based on a Powershell application

This Azure Batch walkthrough creates a Batch account, storage account, package, a pool of VMs, and executes a job with multiple tasks to generate decimal representations of Mersenne prime numbers. If you just want to do the walkthrough, do the following steps; if you also want the why and how, read the background information as well.

Make sure you have access to an Azure subscription. You will need to type credentials of at least Contributor level.
Install the latest Azure Powershell modules.
Download the two toplevel scripts Create-BatchAccountMersenne.ps1 and Create-BatchJobsAndTasks.ps1.
Run Create-BatchAccountMersenne.ps1.
Open the Azure Portal, locate for the (default) resource group named "rg-batch-walkthrough", and inspect it a bit.
Run Create-BatchJobsAndTasks.ps1. This might take a while to complete.
Inspect the jobs, tasks, and task output.
Locate the Storage Account, File Server, a share called (default) "mersenneshare", and you should have the Mersenne primes right there.

This ends the walkthrough steps. Next up is a discussion of the Powershell code needed to create the Azure Batch infrastructure.

Creating the Batch Account using Powershell

The full script to create the Azure Batch walkthrough account is Create-BatchAccountMersenne.ps1 . The code below is straight from the script, stripped from comments and additions. You can either execute the script itself after downloading, or run the snippets below one-by-one. First, log on and select the appropriate subscription.

Add-AzureRmAccount
# select a subscription using select-azurermsubscription, if needed.

We need some definitions to lock down the configuration. They mostly speak for themselves. Important to note is that the name Mersenne is used everywhere and should not be changed without a thorough investigation. The variable PackageURL is a URL to download the actual (zip) package containing the code to be run. In this case, the application code is Powershell script. Feel free to host the ZIP file wherever you like. The hash definition $poolWindowsVersion is the human-readable definition of the Windows version to use for the pool VMs.

$ResourceGroupName = "rg-batch-walkthrough"
$Region = "Central US"
$BatchAccountNamePrefix = "walkthrough"
$WindowsVersion = "2016"

$Applicationname = "Mersenne"
$PoolName = "Pool1"
$ShareName ="mersenneshare"
$Nodecount = 2
$PackageURL = "https://github.com/wkasdorp/Azure-Batch/raw/master/ZIP/MersenneV1.zip"
$poolWindowsVersion = @{
    "2012"     = 3
    "2012R2"   = 4
    "2016"     = 5
}

The next bit is interesting: we need some worldwide unique names, preferably without having the user specify them or trying variations until we get a good one. For this, we use a function that takes the ID of a resource group and mangles this to a semi-random number. Take a look at an earlier post that explains this function in detail.

function Get-LowerCaseUniqueID ([string]$id, $length=8)
{
    $hashArray = (New-Object System.Security.Cryptography.SHA512Managed).ComputeHash($id.ToCharArray())
    -join ($hashArray[1..$length] | ForEach-Object { [char]($_ % 26 + [byte][char]'a') })
}

The business part starts here. First, we need a Resource Group. Because I anticipate that this part of the code may need to be re-run a couple of times, I made it restartable (meaning, do the smart thing if the Resource Group exists):

$ResourceGroup = $null
$ResourceGroup = Get-AzureRmResourceGroup -Name $ResourceGroupName -ErrorAction SilentlyContinue
if ($ResourceGroup -eq $null)
{
    $ResourceGroup = New-AzureRmResourceGroup –Name $ResourceGroupName -Location $Region -ErrorAction Stop
}

An Azure Batch accounts needs a Storage Account to store packages and data, so we first create the Storage Account, and then provision the Batch Account with a reference to the Storage Account. We also create an SMB share in the Storage Account, because an SMB share is easy to write to from Powershell running in the pool VMs. This part is also restartable, which is why we end up with explicitly retrieving the Azure Batch context. Note the use of function Get-LowerCaseUniqueID to determine the names of the Batch and Storage accounts.

$BatchAccountName = $BatchAccountNamePrefix + (Get-LowerCaseUniqueID -id $ResourceGroup.ResourceId)
$StorageAccountName = "sa$($BatchAccountName)"
$BatchAccount = $null
$BatchAccount = Get-AzureRmBatchAccount –AccountName $BatchAccountName –ResourceGroupName $ResourceGroupName -ErrorAction SilentlyContinue
if ($BatchAccount -eq $null)
{
    $StorageAccount = New-AzureRmStorageAccount -ResourceGroupName $ResourceGroupName -Name $StorageAccountName -SkuName Standard_LRS -Location $Region -Kind Storage
    $Share = New-AzureStorageShare -Context $StorageAccount.Context -Name $ShareName
    $BatchAccount = New-AzureRmBatchAccount –AccountName $BatchAccountName –Location $Region –ResourceGroupName $ResourceGroupName -AutoStorageAccountId $StorageAccount.Id
}
$BatchContext = Get-AzureRmBatchAccountKeys -AccountName $BatchAccountName -ResourceGroupName $ResourceGroupName

Next up is to create an application generating some data. As mentioned, the application is pre-packaged as a ZIP file containing two Powershell scripts. We download this into a temporary file, and then generate a new Azure Batch application definition in the existing account. To make life easier for application management, we explicitly define a default version ("1.0").

$tempfile = [System.IO.Path]::GetTempFileName() | Rename-Item -NewName { $_ -replace 'tmp$', 'zip' } –PassThru
Invoke-WebRequest -Uri $PackageURL -OutFile $tempfile 
New-AzureRmBatchApplication -AccountName $BatchAccountName -ResourceGroupName $ResourceGroupName -ApplicationId $applicationname
New-AzureRmBatchApplicationPackage -AccountName $BatchAccountName -ResourceGroupName $ResourceGroupName -ApplicationId $applicationname `
    -ApplicationVersion "1.0" -Format zip -FilePath $tempfile 
Set-AzureRmBatchApplication -AccountName $BatchAccountName -ResourceGroupName $ResourceGroupName -ApplicationId $applicationname -DefaultVersion "1.0"

Finally, we create a pool of VMs that will be used to run the package. I made a number of design choices here.

I used a cloud service (PaaS) VM. This is faster to deploy, but more limited in functionality. For instance, we are limited to Windows although Azure Batch supports Linux as well.
The pool has dedicated VMs which run as long as the pool exists. Again, this is faster for testing, but also more expensive. The alternative is to use low priority nodes that get provisioned when needed. Also, it is possible to increase or decrease the number of nodes in pool.

$appPackageReference = New-Object Microsoft.Azure.Commands.Batch.Models.PSApplicationPackageReference
$appPackageReference.ApplicationId = $applicationname
$appPackageReference.Version = "1.0"
$PoolConfig = New-Object -TypeName "Microsoft.Azure.Commands.Batch.Models.PSCloudServiceConfiguration" -ArgumentList @($poolWindowsVersion[$WindowsVersion],"*")
New-AzureBatchPool -Id $PoolName -VirtualMachineSize "Small" -CloudServiceConfiguration $PoolConfig `
    -BatchContext $BatchContext -ApplicationPackageReferences $appPackageReference -TargetDedicatedComputeNodes $Nodecount

After executing this code, the Azure Fabric provisions two VMs in the pool. This should take about 10-20 minutes. After the nodes have fully initialized you should have the Azure Portal looking something like the following screenshot. There are zero task states, obviously, and two running nodes. At this point, we are ready to run the application package, Mersenne. Doing this is slightly involved because we need to create a Batch Job, with a task for each Mersenne number to be calculated. The command line used to start the Powershell script in the VMs needs to be constructed as well. This is the subject of the next post in this short series.

Application code: decimal version of Mersenne Prime Numbers

This section really is nothing more than background information on the sample application. All you need to know is that it generates prime numbers. The sample application for for this walk-through is a very simple but CPU-intensive calculation. Throughout known history people have searched for ever larger prime numbers, which are numbers that have no divisors except themselves. You know: 2, 3, 5, 7, 11, 13, 17, ... etc. There is an infinity of them. Currently, the largest known prime numbers are Mersenne numbers. I explained this a little bit in the readme.md on Github. The very short summary: a Mersenne number is an exponential of 2, minus one. The exponent is always a prime number. Examples: 2⁷-1 = 127, 2³¹-1 = 2147483647, and the currently (1-3-2018) largest one: 2^77232917-1 = . The Mersenne application packed in the ZIP file MersenneV1.zip simply calculates (some) of these numbers. The following is slightly simplified from the version in the package. You can paste this in a Powershell ISE windows, and it will just work:

# left out argument handling, using Read-Host instead.
$MersenneExponents = @(
    2, 3, 5, 7, 13,
    17, 19, 31, 61, 89,
    107, 127, 521, 607, 1279,
    2203, 2281, 3217, 4253, 4423,
    9689, 9941, 11213, 19937, 21701,
    23209, 44497, 86243, 110503, 132049,
    216091, 756839, 859433, 1257787, 1398269,
    2976221, 3021377, 6972593, 13466917, 20996011,
    24036583, 25964951, 30402457, 32582657, 37156667,
    42643801, 43112609, 57885161, 74207281, 77232917
)

function PrintMersenneDecimal ([int] $n, $width = 80)
{
    $prime = [numerics.biginteger]::pow(2,$n)-1
    $s = $prime.ToString()
    "Mersenne prime 2^$n-1 has $($s.length) digits."
    for ($n=0; $n -lt $s.length; $n += $width)
    {
       $s.Substring($n, [math]::min($width, $s.Length - $n))
    }    
}

$index = $(Read-Host -Prompt "Which Mersenne number to calculate? (0-$($MersenneExponents.count-1))") 
PrintMersenneDecimal -n $MersenneExponents[$index]

Simple enough, I guess. The part doing the actual work is a oneliner based on a .NET library: [numerics.biginteger]::pow(2,$n)-1. Let's move on the final part, a glue script called generate_decimal_mersenne_and_upload.ps1. Its purpose is to serve as an interface between the Azure Batch infrastructure and the code doing the actual work. It reads the arguments passed to the Azure Batch Tasks, and takes care of writing the resulting data back to the SMB share defined on the Storage Account. Simplified version:

[CmdletBinding()]
Param
(
    [int] $index,
    [string] $uncpath,
    [string] $account,
    [string] $SaKey
)

$batchshared = $env:AZ_BATCH_NODE_SHARED_DIR
$batchwd = $env:AZ_BATCH_TASK_WORKING_DIR
$outfile =  (Join-Path $batchwd "Mersenne-$($index).txt")

$generateMersenne = "$env:AZ_BATCH_APP_PACKAGE_MERSENNE\calculate_print_mersenne_primes.ps1"

&$generateMersenne -index $index > $outfile

New-SmbMapping -LocalPath z: -RemotePath $uncpath -UserName $account -Password $SaKey -Persistent $false 
Copy-Item $outfile z:

The main points to note are the mandatory use of environment variables for file and directory paths, the fact that some variable names are based on the actual package name (Mersenne), and the literal reference to the script calculating the Mersenne primes: calculate_print_mersenne_primes.ps1.

Next step: submit a job to the Azure Batch account.

In this post I have shown you step-by-step how to create an Azure Batch account using Powershell. The next step is to create a batch job to run the Mersenne test application on the pool nodes. For this, continue to part 2:

Azure Batch for the IT Pro - Part 2

Download the original Active Directory Branch Office Deployment Guide

Willem Kasdorp — Tue, 09 Jan 2018 23:22:58 GMT

During the great Windows Server 2003 content purge on TechNet in the summer of 2016 a lot of valuable documentation was lost. Part of it was recovered in the infamously huge PDF download with 2003 support content, and other content was ported to the new documentation site on https://docs.microsoft.com, but the rest was just gone. However, us AD people are still missing a historic and still relevant piece of documentation: the AD Branch Office Deployment Guide, originally written for Windows 2000 and later updated for Windows Server 2003. This document outlines the design of an AD with one or more central locations, and lots of outlying branch offices. The DNS configuration described in this removed document was (and is) very relevant.

Guess what? I saved the original download and have it right here on my laptop. Because we (PFE and Support Engineers) still get questions about it, I decided to put it back up again as an informal blog resource. The original download was a .EXE, which was the modern thing to do at the time. It contains two other .EXE files, again doing nothing else but extracting stuff. However, a .EXE is not allowed on WordPress, so I had to repackage it as a .ZIP. Enjoy:

Download the AD Branch Office Deployment Guide for Windows Server 2003 (MD5: 2B33F92B00D8094D27B4A3542BBBFB5C): adbodg03.

Get-UniqueString: generate unique ID for Azure Deployments

Willem Kasdorp — Sat, 23 Dec 2017 22:35:47 GMT

When deploying resources to Azure, you sometimes need to generate a world-wide unique name. Examples of these are DNS names, storage account names, Azure Batch account names, etc. Some of these names have additional requirements. For instance, storage account names must be all lowercase with a length of 3 to 24 letters. How do you reliably generate such names? With manual input this is not so hard. The user just needs to retry until he gets it right. With automation, the problem gets a little more difficult. Let's take an example. You want to generate VM, with a public IP address including a DNS name. The user should tell you the name of the resource group, the VM name, and nothing more. The configuration has more parameters than that, so you need to make up some parameters on the fly. With an existing resource group name of "rg-adtest" and a VM name of "dc1", the definition of the Public IP address could look like this: [powershell] $publicIp = New-AzureRmPublicIpAddress -Name "dc1-pub-ip" -ResourceGroupName "rg-adtest" ` -AllocationMethod Static -DomainNameLabel "" -Location "west europe" [/powershell] The problem is the -DomainNameLabel parameter. Whatever you put here, it must be worldwide unique or the deployment will fail. Just using the string "dc1" is not going to work, obviously (yes, I tried). One solution is to use a random generator to create a fixed-length lowercase random string, something like this: [powershell] "dc1-" + -join ((97..122) | Get-Random -Count 13 | % {[char]$_}) [/powershell] But this has another problem; the code is not repeatable. Each time it runs, the name for the IP address is going to come out different. So how do we solve this? How do we get a string that is random enough to be worldwide unique, but is the same every time the code is executed? When you use Azure ARM templates for deployment, this is easy. You use the UniqueString function, which generates a 13-character unique string based on some other input string. It's traditional to use the ID of the resource group, which is a string containing the world-wide unique Azure Subscription GUID, as well as the name of the resource group. Using this, you get the following results:

Deploy to the same resource group, you get the same name.
Deploy to a different resource group in the same subscription, you get a different name.
Deploy to the same resource group name, but in a different subscription, you get a different name.

Useful stuff. There is no direct PowerShell equivalent for this, so I decided to put one together in the form of a new function called Get-UniqueString. It's short, but a little bit arcane if encryption technology is not your thing. Take a look: [powershell] function Get-UniqueString ([string]$id, $length=13) { $hashArray = (new-object System.Security.Cryptography.SHA512Managed).ComputeHash($id.ToCharArray()) -join ($hashArray[1..$length] | ForEach-Object { [char]($_ % 26 + [byte][char]'a') }) } [/powershell] It does something very simple: it takes a string as input, and generates something that looks random, but really isn't: for the same string, the same output is generated, meaning that the output is deterministic. Any different input string: different output. From the input string it calculates a 512 bit hash value (64 characters) using a cryptographically strong algorithm, which ensures that the result "looks" random. The result is translated back to a string with all lowercase characters, with a default string length of 13 just like the corresponding ARM function. Examples: [powershell] PS C:\> Get-UniqueString -id "123" xjtclbzhdshll PS C:\> Get-UniqueString -id "1234" ehdsuphgcqoyk PS C:\> Get-UniqueString -id $(Get-AzureRmResourceGroup "rg-adtest").ResourceID dbewhnjvchmfk [/powershell] Putting it together, the generation of the IP Address could be: [powershell] $dnsname = "dc1-$(Get-UniqueString -id $(Get-AzureRmResourceGroup "rg-adtest").ResourceID)" $publicIp = New-AzureRmPublicIpAddress -Name "dc1-pub-ip" -ResourceGroupName "rg-adtest" ` -AllocationMethod Static -DomainNameLabel $dnsname -Location "west europe" # what DNS name did we get? $dnsname dc1-dbewhnjvchmfk [/powershell] Having a unique but deterministic random string is a powerful trick. It enables scenarios like these:

Use the Resource Group ID to generate a name for a storage account. This way you can add resources without having to specify the name of the storage account. At the same time, the name will be different for another resource group or subscription.
Use the Subscription ID for stable resources that should be shared by all. Key Vaults, VNETs, ...

A word of warning to wrap this up: don't fall into the temptation of using these functions to generate a password or connection string. You will find this trick on various blog posts discussing ARM templates. The output string looks random, but it really isn't, and these values can easily be regenerated by anyone having read access to your subscription, as discussed in my previous post. For a practical application of Get-UniqueString, see these posts. The first post creates some Azure Resources such as a Batch Account and Storage Account which must be world-wide unique, and the second one derives the names of these accounts straight from its Resource Group.

Best practices for a stable AGPM deployment

Willem Kasdorp — Mon, 11 Dec 2017 16:02:47 GMT

Over the years I have worked a lot with Advanced Group Policy Management (AGPM), our solution for change management of Group Policy. This small tool is part of Microsoft Desktop Optimization Pack (MDOP). AGPM has always flown under the radar, but is deployed in surprisingly many enterprises. By reviewing and troubleshooting some of these deployments I have picked up what works well, and what does not. In this blog I want to share some pointers on improving AGPM deployments.

AGPM does not use the PDC

Let's start with a big one. If you know your group policy, you will be aware of the special role of the PDC role: it is targeted as the default Domain Controller for editing Group Policy. This means that when you start the GPMC or the GP Editor, they will look at the PDC for reading and writing GPO. Most (not all) Client-Side Extensions (CSE) such as the Admin Templates, Security and others also play nice with this convention. However, AGPM does not honor the convention of using the PDC emulator when reading or writing GPOs. Instead, it relies on the default DCLocator behavior and will simply select the closest DC, usually one in its own site. Problem? You bet. Suppose you are editing a GPO that is managed with AGPM. Also, assume that the AGPM server is deployed in a site with significant replication latency to the site with the PDC. You start the edit cycle with a check out of a GPO, so AGPM will push a copy of the GPO to its own site. If you try to edit the GPO immediately after the checking out, it will not have replicated to the PDC yet, or perhaps only partially. You might see no GPO at all, or an empty one, or worst case: a corrupted one! Variations of this theme happen on check-in or GPO redeployment. Fortunately, we have an obvious but partial solution: make sure the AGPM server shares the same site as the PDC. This minimizes replication latency to the PDC. Why is it partial? Because AGPM might still pick another DC in the same site instead of the PDC itself, leaving room for replication latency issues. Still, it's much better than having the AGPM server associated with another site. But what if the AGPM server is physically in another location than the PDC emulator? Then we can still solve it by using a little trick: we use a /32 subnet to hardwire the AGPM server to the site of the PDC:

In AD Sites & Services, create a new subnet with the IP address of the AGPM server.
As a netmask suffix, use /32
Link the subnet to the site containing the PDC emulator
Add a comment to the effect that you are hardwiring the AGPM server to the PDC site.

It might look as follows:

AGPM does not like FRS

We have seen that AGPM depends on SYSVOL replication, and that the faster replication is, the better your AGPM experience will be. Actually, AGPM depends on AD replication as well because a GPO has both an AD and a SYSVOL component but let's leave that be for now. The old-school FRS mechanism is dead slow compared to DFS-R. So, if you still have your Domain running on FRS, it's time to migrate it to DFS-R, as I have blogged before. However, even if you are running DFS-R already, you still may have occasional errors and problems. Due to the rate of change that AGPM pushes on SYSVOL when checking out or deploying, DFS-R might run into problems. Most of these have been fixed, but to benefit from these fixes you still need to deploy the latest versions of the DFS-R binaries. If you are on Windows Server 2016 or 2012 R2 and have kept up with update rollups, you are probably fine. But, if you are on Windows Server 2008 R2 SP1 with original binaries, you really want to upgrade. For this reason and many other fixes, please deploy the Convenience rollup update for Windows 7 SP1 and Windows Server 2008 R2 SP1.

AGPM likes to control all GPOs

The biggest reason why AGPM deployments fail is actually not a technical one. It's process related. Imagine that you have 500 GPOs, of which 450 are controlled by AGPM, and 50 are not. You need to edit a particular one, but before you can do that you first need to know if you need to go through AGPM or not. The only way to know is look in AGPM, even if the GPO is not controlled there. Admins tend to get tired of this confusion pretty fast, and resort to editing GPOs outside of AGPM all the time. This causes the AGPM archive to become outdated compared to production. This is a significant hazard in itself: what happens if someone deploys an outdated GPO from the AGPM archive, and overwrites the current production GPO? You need only one or two incidents like that before someone decides to get rid of APGM altogether. Again, a simple solution: get buy-in to manage really all GPO's in the domain using AGPM. If you can't get this buy-in, don't deploy AGPM. Believe me, I have seen enough of this. The only two policies that you might want to leave out for security reasons are the two built-in policies: Default Domain Policy, and Default Domain Controller Policy. Having all GPOs in AGPM has an additional advantage: it can be used as the single backup solution for GPO, including versioning, history, and change rollback. This does require that the AGPM server itself has a backup.

Locking down Domain Admins is good for AGPM

By default, Domain Admins can edit GPOs without using AGPM. While convenient, this can cause a lot of problems as outlined in the previous section: outdated archives. For this reason, I usually recommend giving Domain Admins read-only access on all GPOs. Of course, this means nothing from a security perspective because Domain Admins can reset the permissions and do as they please. However, it prevents accidents and would take a deliberate action from an admin to edit a GPO outside of AGPM. A PFE colleague has blogged about this before:

https://blogs.technet.microsoft.com/askds/2011/06/21/forcing-domain-admins-to-use-agpm-but-not-really/

Have a look, and take from it what you need. It shows how to stop Domain Admins from accidentally creating and deleting GPOs as well. Personally, I'd not go as far to stop Domain Admins from linking GPOs to OUs or to the domain.

AGPM should always have the latest version of all GPOs

As you have seen, you are asking for problems when a currently deployed GPO is more recent than the AGPM archive version; if you would deploy the GPO from the archive, it would overwrite the production version with old data. There are multiple reasons why the AGPM archive might be outdated:

You changed the GPO security filtering
You added or removed a WMI filter
You did an emergency change outside of AGPM
You had to restore the AGPM archive
You restored a GPO outside of AGPM
etc.

There are two questions here:

How do I know that the AGPM archive is outdated? Answer: have a look at an older blog of mine: Find out if your AGPM archive needs updating. This blog has a reference to a script that will show you exactly which GPOs are at risk by comparing timestamps of the archive with AD.
How do I recover and make sure the AGPM archive is up to date? Answer: for each affected GPO, you do an Import from Production. See the following screenshot in case you have not done this before.

Keep up with the latest AGPM version

AGPM is in maintenance mode, and bugs do get fixed. The current version (December 2017) is AGPM 4.0, Service Pack 3, Hotfix Level 3. While AGPM has both a client and a server part, it's the server part that does the work. So please prioritize the AGPM service when upgrading AGPM. If AGPM is not doing what it should, execute an in-place upgrade first. Do read the documentation; you may have to install a newer version of the .NET platform. Specifically, if you want to use AGPM to control GPOs for Windows 10 or for Windows Server 2016, you do need to run the MDOP 2016 version or later.

Do you have plaintext passwords in your Azure deployments?

Willem Kasdorp — Sat, 02 Dec 2017 21:45:29 GMT

If you are developing deployments for Azure you will encounter situations where you need to use passwords and other data that needs to stay hidden. Azure has plenty of facilities for this, but sometimes people can be tempted to take shortcuts. So, for one of the projects I'm involved in there was a suspicion that not everyone had been diligent. I wrote a quick Powershell script to walk though all deployments looking for parameters named *secret* or *password*, and having a type not called "SecureString". This script iterates over all subscriptions that the current credentials have permissions for, and over all resource groups. It would be easy enough to modify the script for more selective filtering. [powershell] Get-AzureRmSubscription | ForEach-Object { $subscriptionname = $_.Name $_ | Select-AzureRmSubscription | Out-Null Write-Verbose "- processing subscription $subscriptionname" Get-AzureRmResourceGroup | ForEach-Object { $rgname = $_.ResourceGroupName Write-Verbose "-- query deployments for RG $rgname" $deployments = Get-AzureRmResourceGroupDeployment -ResourceGroupName $rgname $deployments | ForEach-Object { $deployment = $_ if ($keynames = $deployment.Parameters.Keys -match "(password)|(secret)") { $keynames | ForEach-Object { $type = $deployment.Parameters.$_.Type $value = $deployment.Parameters.$_.Value if ($type -ne "SecureString") { Write-Verbose "--- found non-secure password field(s) among keys $($keynames -join ',')" [pscustomobject] @{ Subscription = $subscriptionname RG = $rgname Deploymentname = $deployment.DeploymentName PasswordFieldName = $_ PasswordFieldType = $type PasswordFieldValue = $value TimeStamp = $deployment.Timestamp } } } } } } } [/powershell] The script assumes that authentication has already happened; if not, just run add-azurermaccount first. The output is an object that you could pipe to Out-Gridview or Export-CSV. Fair warning: the output of this script could contain plaintext passwords...