Well...actually...

Workflow Parallell and Sequence....for the control freak

Joel Vickery, PFE — Thu, 23 Aug 2018 04:24:41 GMT

Following up on my last post on PowerShell WorkFlows , I thought I would take a moment to dive a little deeper into a topic that I personally could not find a lot of good details on, at least in a way that made sense to me: How the Parallel and Sequence functions work within workflows.

Sequence

In the movie "As Good As It Gets", Jack Nicholson plays a character that battles with OCD. One of his character quirks is that he compulsively sorts his candy by color into separate jars. If you sympathize with that character or behavior, you're going to the love the control that the Sequence script block provides within a WorkFlow. Sequence is useful for situations where you have a list of commands to run that have dependencies on the order in which they run (i.e. task #1 needs to run to completion before task #2). The workflow script below basically is passed an array of machines, going through each server (in parallel, note the "-parallel" parameter in the ForEach line) to get the newest 2000 event 4264's (successful logons) from the security event log and returning the count of the event IDs found. Next the event ID 4634's (logoff events) are queried and counted in the same manner.

WorkFlow ParallelSequenceTest
{
    param([Array[]]$Servers)

    ForEach -Parallel -ThrottleLimit 4 ($Server in $Servers)
    {
        Sequence
        {
            $EvntCnt = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4624'}).Count
            Write-Output "$Server $EvntCnt Event 4624's"
            $EvntCnt2 = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4634'}).Count
            Write-Output "$Server $EvntCnt2 Event 4634's"
        }
    }
}

You will notice that I have a "ForEach -Parallel" statement in the workflow with a "-ThrottleLimit 4" parameter. This limits the WorkFlow to 4 concurrent threads and sets the table for an explanation of how the Sequence statement within the WorkFlow works. Below is an illustration to help visualize how this works, based on a group of 9 servers. As you can see in the script output below, the D4, DC3, DC2, & DC1 get all of the Event 4624's. Then they do the next query of event 4634's. As noted above, the WorkFlow is limited to 4 threads, hence the 4 machine grouping. Now the next group of machines (DC6, DC7, DC8, & DC5) repeat the same process of retrieving the 4624 Events, then moving on to the 4634 events. Since I had 9 machines and limited the WorkFlow to 4 threads at a time, there is 1 lonely machine left to process, my Tools server. True to form, the script processes the event 4624's then does the event 4634's.

Parallel

Since I have a movie theme going, I'm going to use my favorite Michael Keaton movie "Multiplicity" for comparison. In Multiplicity, the lead character creates a clone of himself so that he could do his demanding job while spending time with his family and getting all of his home improvement projects done. The movie gets interesting when the clones start making more clones...you get the picture. Doing a parallel statement within a parallel loop always makes me think of that movie. The Parallel script block is useful for situations where you have a list of commands to run that have no dependencies on the order that they are run. In my example code below. I am querying the Security Event Logs for event 4624 (successful logons) and Event 4634 (successful logouts). With a slight modification, I have changed the processing behavior of the WorkFlow by changing the Sequence script block to a Parallel script block below:

WorkFlow ParallelParallelTest
{
    param([Array[]]$Servers)

    ForEach -Parallel -ThrottleLimit 4 ($Server in $Servers)
    {
        Parallel
        {
            $EvntCnt = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4624'}).Count
            Write-Output "$Server $EvntCnt Event 4624's"
            $EvntCnt2 = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4634'}).Count
            Write-Output "$Server $EvntCnt2 Event 4634's"
        }
    }
}

When this block of code runs. the behavior will be quite different. To stay consistent for comparison's sake, I'll stick with 4 concurrent threads in the ThrottleLimit parameter and use the same 9 machines for processing. Below is an overview of how the Parallel script block executes in this scenario. As described above, each of the first 4 servers processes the event logs to return the event 4624 and the 4634's at the same time, then the next group of servers processes the events, finally leaving server #9 to process the events.

Sequencellel/Parallence?

Now it gets really weird...what would happen if we mix the Parallel and Sequence script blocks? Let's find out. I'm adding a new couple of Event ID's to the script, namely events 4769 (Kerberos service ticket requested) and 4770 (Kerberos ticket renewed). The Sequence script block is the same as before, collecting the 4624/4634 events. In addition, in a parallel script block, we will grab all of the 4679/4770 events. If I am going to continue the movie theme for this script...."Primer" definitely comes to mind. If you've never seen it, go watch it, you'll see what I mean.

WorkFlow ParallenceTest
{
    param([Array[]]$Servers)

    ForEach -Parallel -ThrottleLimit 4 ($Server in $Servers)
    {
        Sequence
        {
            $EvntCnt = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4624'}).Count
            Write-Output "$Server $EvntCnt Event 4624's"
            $EvntCnt2 = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4634'}).Count
            Write-Output "$Server $EvntCnt2 Event 4634's"
        }

        Parallel
        {
            $EvntCnt = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4769'}).Count
            Write-Output "$Server $EvntCnt Event 4769's"
            $EvntCnt2 = (Get-EventLog -LogName "Security" -Newest 2000 -ComputerName $Server| `
            Where-Object {$_.EventID -eq '4770'}).Count
            Write-Output "$Server $EvntCnt2 Event 4770's"
        }
    }
}

Below is the script output with Sequence and Parallel script blocks. Notice that the first group of machines (DC4, DC3, DC2, & DC1) process the 4624's then the 4634's but for some reason DC1 takes longer than the others to return the 4634. Before DC1 can return it's 4634 into the output, DC4 has moved on to the Parallel script block and returned it's 4769 & 4770 event counts. The second group of threads (DC6, DC7, DC8, & DC5) now start processing. DC 5 repeats the same issue seen with DC1 in the first group with DC6 moving on to the parallel script block before it can return it's 4634. The Tools server, last but not least, finishes up the processing with the last thread. As you can see, the output of this would be confusing to handle if you expected it to come back in a defined order or grouped together, no candy sorting here. It's important to understand the processing behavior of these code blocks so you know the dependencies and how the output may be returned since it is different between the two script blocks, and definitely different with Sequence and Parallel mixed up. Each one has it's place in the world and hopefully you are now armed with the knowledge to know when and how to use these to solve you toughest problems.

Just go with the flow....WorkFlow that is

Joel Vickery, PFE — Thu, 17 May 2018 04:35:12 GMT

Following up on my original post Parallel Processing with jobs in PowerShell – Juggling multiple chainsaws, I wanted to go into another method of running parallel processes in PowerShell, namely WorkFlows. WorkFlow is very similar to using the Start-Job/-asJob functionality but it has some distinct advantages that are fully covered in many other blog postings on TechNet. I'll mention them quickly below, but if you want deeper detail, see the When Windows PowerShell Met WorkFlow blog posting on MSDN. Finally, I wanted to focus on how PowerShell handles the activity of Workflow from a process/thread perspective so that you can factor in the way that processing workflows works into your decision to use it over other parallel and multi-threading methods. I started this set of posts with the thought "I wonder which one of these methods is the fastest?" so I will include that information as well.

The Great Parts

(Not) Managing the Queue

WorkFlow takes some of the overhead of managing the jobs away. If you have worked with PowerShell Jobs, it doesn't take long to realize that you have a lot of work to do to manage the submission, monitoring, retrieval, and removal of the jobs.

Unique Processing Control

WorkFlow also adds some unique controls for how commands within the workflow execute:

Parallel

Commands that are contained within a Parallel block will run concurrently, meaning that each command does not wait until it is complete before running the next command in the Parallel Block.

Sequence

Commands within a Sequence block will execute each command and wait for the command to complete before running the next command.

Checkpoints

Workflow scripts that are interrupted can be resumed even through a reboot.

The Tough Parts

Restrictions

All good things come at a price. With WorkFlows, there are restrictions that you need to be aware of that take this from a simple exercise to an intermediate-level development effort. I struggled with issues like changing parameters and scoping of variables as well as the lack of support for invoke-command. Like any learning experience, once you get the hang of it, you'll be cranking out WorkFlow code like second nature.

Debugging

Debugging WorkFlows in earlier releases of PowerShell was a challenge since the PowerShell_ISE would lose insight into the processes when they would execute in parallel. This was corrected in PowerShell v4.0, which allows full control of the code within the WorkFlow block, to include setting breakpoints.

Example Script

The script below queries AD for all computers and then, using a WorkFlow, gets the most recent 4000 event 4624 events from the security event log. Since I am using the "-parallel" switch on the foreach loop, querying each of the computers will happen concurrently. I will be adding more posts beyond this one and will use the exact same basic script in the future posts, just wrapped in a different parallel processing wrapper.

workflow Test-WF 

{

     param([array[]]$ServerList)
     $ReturnArr = @()

     foreach -parallel -ThrottleLimit 10 ($Server in $ServerList)
     {
         $returnName = $Server
         $strCompName = $Server.Name
         $Count = InlineScript { (Get-Eventlog Security -ComputerName $Using:returnName -Newest 4000| '
         Where-Object {$_.EventID -eq '4624'}).count}
         $Workflow:ReturnArr += "$returnName,$Count"
     }
     $ReturnArr
 }

$arrComputers = (get-adcomputer -filter * -server dc1.contoso.com:3268).Name

Get-date
$Stats = Measure-Command -Expression {$OutTest = Test-WF -ServerList $arrComputers}
Get-date
$outTest
$Stats

Process Analysis

I have to admit that I made a mistake here, but it turned out to be a good thing since it pointed out an efficiency and performance issue that you can encounter depending how you implement your script. My initial script had an InlineScript Activity on the line of code that queries the event log and that turned out to be an important factor in how efficient and fast this script is. To get a good idea of what is happening behind the scenes, I'm using ProcMon from the Sysinternals Suite to record the process tree that results from the WorkFlow. In my small test lab environment, I have 8 domain controllers and a general tools server, so 9 machines that are returned from the query against AD and then will each be contacted to parse the event logs. In the screen capture above, you will notice that the initial script was started in the PowerShell_ISE, which then created several additional PowerShell.exe processes, each with their own Process ID. In the screen capture below from ProcMon, you can see the PowerShell_ISE.exe creating the final Processes and worker threads to go forth and get the information that we have requested. It's becoming very clear through this capture that WorkFlows, much like Start-Jobs/-asJob are "multi-process", not truly multi-threaded. It turns out that this is due to the fact that I used the InlineScript activity, which run in their own process. Just something to keep in mind if resources are an issue and you cannot afford the overhead of multiple PowerShell.exe processes running. Finally, in the screen capture below, the newly created processes are querying the security event logs in parallel. To be thorough, I went back and modified the script without the use of the InlineScript activity, instead using the direct Get-EventLog call against the remote servers. I changed the line: $Count = InlineScript { (Get-Eventlog Security -ComputerName $Using:returnName -Newest 4000| ' Where-Object {$_.EventID -eq '4624'}).count} to: $Count =(Get-Eventlog Security -ComputerName $Using:returnName -Newest 4000| ' Where-Object {$_.EventID -eq '4624'}).count Now I am seeing the single PowerShell.exe with multiple threads that I was expecting.

Performance

To give a quick frame of reference, I ran a regular sequential ForEach loop against all of these same domain controllers and the sequential PowerShell loop took 1 minute and 23 seconds to complete the retrieval of events from the DCs. In the output below, you can see that WorkFlow with InlineScript beats that time by a considerable margin at 39 seconds. Performance took a hit with the InlineScript activity removes when compared to the multi-process, dropping to 1 minute and 17 seconds. It was still faster than the sequential loop but not by a lot.

Conclusion

As you start using these parallel capabilities in PowerShell, it's important to think about the correct use case for the particular parallel processing capability. For WorkFlows using the InlineScript activity, you have to make sure that you have the resources to handle having multiple full PowerShell.exe environments running, including memory and CPU utilization. Without InlineScript, resource consumption is less of an issue and the script will run in a true multithreading mode. I tend to use Workflows when I have a decent amount of time to create a workable script, and I need the performance increase, and I know I'll be collecting dividends on the time spent when compared to a simple sequential loop.

Parallel Processing with jobs in PowerShell - Juggling multiple chainsaws

Joel Vickery, PFE — Tue, 20 Jun 2017 18:07:52 GMT

I have to be honest, back in 2010 I was firmly entrenched in VBScript and had no interest in learning anything new. Then fate threw me a curve ball when I was presented with a task to fix over 9,000 workstations that had lost their "parent" antivirus server. Each machine needed to have the exact same set of commands executed on it, a new file copied and service restarted, to fix them so I created a quick VBScript to connect to each machine and perform the fix actions. The only bad part about this script was that it took days to run against the machines, processing each one sequentially in a For loop before moving on to the next. I did some research on multi-threading with VBScript and kept getting results that referred to PowerShell "jobs", which gave me the motivation to start learning PowerShell. For those of you who come from a Unix or Linux background, jobs will be a familiar concept since you can background any command by placing an "&" at the end of the command. PowerShell Cmdlets sometimes include the "-AsJob" parameter that will allow the command to submitted in the background, releasing the console to be used for other commands. Alternatively, you can use the Start-Job CmdLet to run commands as Jobs. This also means that you can submit multiple jobs in the background and continue processing without waiting for each job to complete. Keep in mind that there are limits to everything so keep your machine's resource consumption in mind as you test this concept out. Submitting a large number of background commands could be resource intensive. Below are two different ways to do a WMI Query as a job: Get-WMIObject Win32_OperatingSystem -AsJob or Start-Job {Get-WMIObject Win32_OperatingSystem} These jobs do all of their processing in the background and store the output until you receive them. You can check on the status of a job by running Get-Job, the status is in the "State" column, which will show if the command is Running, Completed, or Failed. Also notice the HasMoreData column. This indicates that there is output to be retrieved from the job. In the example output below, notice that the job is still running. Id Name PSJobTypeName State HasMoreData Location Command 1 Job1 WmiJob Running True localhost get-wmiobject win32_op... After the job has been kicked off, you can check on the status of the job by running the Get-Job command, noting the State and HasMoreData values. The State will change to Completed when the job has finished and the HasMoreData value will indicate if there is output. Get-Job Id Name PSJobTypeName State HasMoreData Location Command 1 Job1 WmiJob Completed True localhost get-wmiobject win32_op... Once the job has completed, you can use the Receive-Job cmdLet to get the data from the command. Receive-Job -Id 1 <----- NOTE, you could also do -Name Job1 here, either will work The output of the command is now delivered to the console: SystemDirectory : C:\Windows\system32 Organization: Contoso Build Number: 15063 Registered User: ContosoUser SerialNumber: Version: 10.0.15063 Note that the HasMoreData value has now changed to False after running the Receive-Job command: Get-Job Id Name PSJobTypeName State HasMoreData Location Command 1 Job1 WmiJob Completed False localhost get-wmiobject win32_op... The important thing to remember here is that you have one chance to get the information from the job so make sure that you capture it in a variable if the output needs to be evaluated. $JobOutput = Receive-Job -Id 1 Once you are done getting the job's output, the job will basically hang out there until you remove it by running the Remove-Job Cmdlet. Remove-Job -Id 1 As you can tell, there are a lot of moving parts to this. The PowerShell script that I created to address the antivirus client remediation task had to have a mechanism to control how many jobs I could submit at a time, as well as constant monitoring of the queue to receive input from the completed jobs, remove them, and refill the queue with new jobs. To top it off, once I finished processing the queue of items, I had to write some code to wait for the last batch of jobs was completed to make sure that I received the output from all of them. That is a lot of overhead. Fast forward to today and we have PowerShell WorkFlows that make this process much easier to manage which I'll cover in another post.

Windows 10 IOT Enterprise Update installation "Failure configuring Windows updates. Reverting changes" – How can something so simple be so hard?

Joel Vickery, PFE — Fri, 16 Jun 2017 20:57:37 GMT

Let me answer the question before we get started, nothing is ever as simple as you think it is going to be. In a large organization there are usually lots of different environmental factors that can impact the installation of software or patches. As PFEs and Consultants for Microsoft, we usually have the daunting task of tracking down the needle in the update installation haystack and it seems like we never hit the same issue twice. While helping our customer VDI team with the installation of the latest Cumulative Updates for Windows 10 IOT running on thin client hardware, we ran into a challenging situation. The update would install successfully, however, would fail at the reboot and would roll back the installation of the update, giving the dreaded "Failure configuring Windows updates. Reverting changes. Do not turn off your computer" message. As a system administrator, that message usually translates to "put on a pot of coffee, we're going to be here a while". The very first thing to look at when troubleshooting the installation of Windows Updates is the CBS.log file, located in the C:\Windows\Logs\CBS folder. Reading the CBS.log file can be overwhelming so try to develop some tactics like looking for terms like "failed", "failure", or "error" unless you really just like reading long log files. In our case, the first indicator of failure was a message related to the creation of a temporary file during the update installation: C:\Windows\system32\wbem\mispace.mof (1): error SYNTAX 0X8004402f: Error creating temporary file While obscure, the CBS.log entry above does give us a place to start to narrow down the problem. If you have ever dealt with thin desktop solutions, you know that they usually run a minimal interface and are not configured to retain changes. Their function in life is usually to act as a Window through which you view the virtual desktop. In this case, the hardware manufacturer utilizes a small portion of the system RAM to create an overlay which tracks changes to the protected storage drive of the thin client. The space requirement to install the latest cumulative update was too much for the flash overlay area to hold at only about 500 MB. The solution to this issue was to temporarily modify the %TEMP% and %TMP% SYSTEM environment variables to point them from the overlay RAM Drive (Z:\Temp) to the C:\Temp directory where there is sufficient space. Word to the wise, make sure that you create that directory or you will be fighting this issue for a while longer. If you forget to create the above-mentioned directory, the error message and failure behavior will look exactly the same as it did when pointing to the overlay drive. After making the change to the %TEMP% and %TMP% variables, the update installed with no issues. Many thanks to my teammates Bill Almonroeder and David Morillo for helping track this issue down to resolution. No pot of coffee was required if you were wondering.

Cannot install service account. The provided context did not match the target

Joel Vickery, PFE — Tue, 25 Apr 2017 04:43:25 GMT

I've been doing system administration since roughly 1994 and in that time I've come to realize one thing: making changes to established environments always causes a ripple effect. The impact of changes usually doesn't surface right away, making the process of associating the exact change to the impact days, weeks, or months down the road very difficult. Recently my Microsoft colleague Bill Almonroeder and I were helping our customer implement a Group Managed Service Account (gMSA) to increase security and lower the administrative overhead on our service accounts. We were methodical in the steps to properly create the account and waited a full weekend between the time that we created the account in active directory. Sounds good so far, right? In complex compute environments where there are many moving parts, things rarely go as planned and this day was no exception. When we attempted to perform the installation of the gMSA on Windows Server, we received the error message: Install-ADServiceAccount : Cannot Install service account. Error Message: 'The provided context did not match the target' Hmmm.."The provided context did not match the target"...not exactly overflowing with relevant information. Maybe the Test-ADServiceAccount cmdlet will have some information that we are not seeing here. The Test-ADServiceAccount cmdlet gives us a little more to go on: WARNING: Test failed for Managed Service Account SQLServerGMSA. If standalone Managed Service Account, the account is linked to another computer object in the Active Directory. If group Managed Service Account, either this computer does not have permission to use the group MSA or this computer does not support all the Kerberos encryption types required for the gMSA. See the MSA operational log for more information. We verified that the account had been created properly and did not suffer from the linked to another computer object condition mentioned first in the message, which would be applicable in a Managed Service Account (MSA) scenario since that type of account can only be used with a single computer. Our account was created as a gMSA, and we validated that the group that had been associated with the GMSA was properly configured and contained both servers on which we intended to use the gMSA. Permissions were verified, all is well. What is this about the Kerberos encryption, we're not experiencing any other Kerberos-related issues. Time to look at the event logs. A quick look at the Security-Netlogon event log yielded a couple of clues. There were event IDs 9001 and 9002 being logged when we attempt to test and install the GMSA. The 9002 event doesn't provide much more information than the Install-ADServiceAccount cmdlet: Netlogon failed to add SQLServergMSA as a managed service account to this local machine. The provided context did not match the target Let's look at the 9001: The account SQLServergMSA cannot be used as a managed service account on the local machine because not all of the supported encryption types are supported by the local machine. This event is at least pointing us back at the encryption types, which is also mentioned at the tail end of the Test-ADServiceAccount output. What gives? After dredging through the documentation for gMSAs, I ran across the answer to our issue at: https://technet.microsoft.com/en-us/library/hh831782.aspx : "A managed service account is dependent upon Kerberos supported encryption types. When a client computer authenticates to a server using Kerberos the DC creates a Kerberos service ticket protected with encryption both the DC and server supports. The DC uses the account's msDS-SupportedEncryptionTypes attribute to determine what encryption the server supports and, if there is no attribute, it assumes the client computer does not support stronger encryption types. If the Windows Server 2012 host is configured to not support RC4 then authentication will always fail. For this reason, AES should always be explicitly configured for MSAs" Ah...now it makes sense! We had recently decided to remove the RC4 encryption cipher to tighten security and, until now, had not run into an issue. So many changes like that do not expose problems immediately, you see the effects weeks or months down the road. The exact setting that was impacting us is below, and we verified that RC4 was disabled: Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Security Options -> Network Security: Confiure encryption types allowed for Kerberos Now we're making progress! We'll heed the advice of the documentation and specify the encryption types for the gMSA using the command below: Set-ADServiceAccount -Identity SQLServerGMSA -KerberosEncryptionType AES128,AES256 To verify, we can look at the GMSAs attributes in Active Directory Users and Computers, specifically, the msDS-SupportedEcryptionTypes attribute to verify that it does not contain RC4. Now that the problem has been resolved, the Install-ADServiceAccount works with no errors, hopefully no news is good news. If we run the Test-ADServiceAccount cmdlet, note that it returns "True" now. That looks much better! Our gMSA is finally ready for use.