Edit

about_Thread_Jobs

  • Article

Short description

Provides information about PowerShell thread-based jobs. A thread job is a type of background job that runs a command or expression in a separate thread within the current session process.

Long description

PowerShell concurrently runs commands and scripts through jobs. There are three jobs types provided by PowerShell to support concurrency.

  • RemoteJob - Commands and scripts run in a remote session. For information, see about_Remote_Jobs.
  • BackgroundJob - Commands and scripts run in a separate process on the local machine. For more information, see about_Jobs.
  • PSTaskJob or ThreadJob - Commands and scripts run in a separate thread within the same process on the local machine.

Thread-based jobs are not as robust as remote and background jobs, because they run in the same process on different threads. If one job has a critical error that crashes the process, then all other jobs in the process are terminated.

However, thread-based jobs require less overhead. They don't use the remoting layer or serialization. The result objects are returned as references to live objects in the current session. Without this overhead, thread-based jobs run faster and use fewer resources than the other job types.

Important

The parent session that created the job also monitors the job status and collects pipeline data. The job child process is terminated by the parent process once the job reaches a finished state. If the parent session is terminated, all running child jobs are terminated along with their child processes.

There are two ways work around this situation:

  1. Use Invoke-Command to create jobs that run in disconnected sessions. For more information, see about_Remote_Jobs.
  2. Use Start-Process to create a new process rather than a job. For more information, see Start-Process.

How to start and manage thread-based jobs

There are two ways to start thread-based jobs:

  • Start-ThreadJob - from the ThreadJob module
  • ForEach-Object -Parallel -AsJob - the parallel feature was added in PowerShell 7.0

Use the same Job cmdlets described in about_Jobs to manage thread-based jobs.

Using Start-ThreadJob

The ThreadJob module first shipped with PowerShell 6. It can also be installed from the PowerShell Gallery for Windows PowerShell 5.1.

To start a thread job on the local computer, use the Start-ThreadJob cmdlet with a command or script enclosed in curly braces ({ }).

The following example starts a thread job that runs a Get-Process command on the local computer.

Start-ThreadJob -ScriptBlock { Get-Process }

The Start-ThreadJob command returns a ThreadJob object that represents the running job. The job object contains useful information about the job including its current running status. It collects the results of the job as the results are being generated.

Using ForEach-Object -Parallel -AsJob

PowerShell 7.0 added a new parameter set to the ForEach-Object cmdlet. The new parameters allow you to run script blocks in parallel threads as PowerShell jobs.

You can pipe data to ForEach-Object -Parallel. The data is passed to the script block that is run in parallel. The -AsJob parameter creates jobs objects for each of the parallel threads.

The following command starts a job that contains child jobs for each input value piped to the command. Each child job runs the Write-Output command with a piped input value as the argument.

1..5 | ForEach-Object -Parallel { Write-Output $_ } -AsJob

The ForEach-Object -Parallel command returns a PSTaskJob object that contains child jobs for each piped input value. The job object contains useful information about the child jobs running status. It collects the results of the child jobs as the results are being generated.

How to wait for a job to complete and retrieve job results

You can use PowerShell job cmdlets, such as Wait-Job and Receive-Job to wait for a job to complete and then return all results generated by the job.

The following command starts a thread job that runs a Get-Process command, then waits for the command to complete, and finally returns all data results generated by the command.

Start-ThreadJob -ScriptBlock { Get-Process } | Wait-Job | Receive-Job

The following command starts a job that runs a Write-Output command for each piped input, then waits for all child jobs to complete, and finally returns all data results generated by the child jobs.

1..5 | ForEach-Object -Parallel { Write-Output $_ } -AsJob | Wait-Job | Receive-Job

The Receive-Job cmdlet returns the results of the child jobs.

1
3
2
4
5

Because each child job runs parallel, the order of the generated results is not guaranteed.

Thread job performance

Thread jobs are faster and lighter weight than other types of jobs. But they still have overhead that can be large when compared to work the job is doing.

PowerShell runs commands and script in a session. Only one command or script can run at a time in a session. So when running multiple jobs, each job runs in a separate session. Each session contributes to the overhead.

Thread jobs provide the best performance when the work they perform is greater than the overhead of the session used to run the job. There are two cases for that meet this criteria.

  • Work is compute intensive - Running a script on multiple thread jobs can take advantage of multiple processor cores and complete faster.

  • Work consists of significant waiting - A script that spends time waiting for I/O or remote call results. Running in parallel usually completes quicker than if run sequentially.

(Measure-Command {
    1..1000 | ForEach { Start-ThreadJob { Write-Output "Hello $using:_" } } | Receive-Job -Wait
}).TotalMilliseconds
36860.8226

(Measure-Command {
    1..1000 | ForEach-Object { "Hello: $_" }
}).TotalMilliseconds
7.1975

The first example above shows a foreach loop that creates 1000 thread jobs to do a simple string write. Due to job overhead, it takes over 36 seconds to complete.

The second example runs the ForEach cmdlet to do the same 1000 operations. This time, ForEach-Object runs sequentially, on a single thread, without any job overhead. It completes in a mere 7 milliseconds.

In the following example, up to 5000 entries are collected for 10 separate system logs. Since the script involves reading a number of logs, it makes sense to do the operations in parallel.

$logNames.count
10

Measure-Command {
    $logs = $logNames | ForEach-Object {
        Get-WinEvent -LogName $_ -MaxEvents 5000 2>$null
    }
}

TotalMilliseconds : 252398.4321 (4 minutes 12 seconds)
$logs.Count
50000

The script completes in half the time when the jobs are run in parallel.

Measure-Command {
    $logs = $logNames | ForEach {
        Start-ThreadJob {
            Get-WinEvent -LogName $using:_ -MaxEvents 5000 2>$null
        } -ThrottleLimit 10
    } | Wait-Job | Receive-Job
}

TotalMilliseconds : 115994.3 (1 minute 56 seconds)
$logs.Count
50000

Thread jobs and variables

There are multiple ways to pass values into the thread-based jobs.

Start-ThreadJob can accept variables that are piped to the cmdlet, passed in to the script block via the $using keyword, or passed in via the ArgumentList parameter.

$msg = "Hello"

$msg | Start-ThreadJob { $input | Write-Output } | Wait-Job | Receive-Job

Start-ThreadJob { Write-Output $using:msg } | Wait-Job | Receive-Job

Start-ThreadJob { param ([string] $message) Write-Output $message } -ArgumentList @($msg) |
  Wait-Job | Receive-Job

ForEach-Object -Parallel accepts piped in variables, and variables passed directly to the script block via the $using keyword.

$msg = "Hello"

$msg | ForEach-Object -Parallel { Write-Output $_ } -AsJob | Wait-Job | Receive-Job

1..1 | ForEach-Object -Parallel { Write-Output $using:msg } -AsJob | Wait-Job | Receive-Job

Since thread jobs run in the same process, any variable reference type passed into the job has to be treated carefully. If it is not a thread safe object, then it should never be assigned to, and method and properties should never be invoked on it.

The following example passes a thread-safe .NET ConcurrentDictionary object to all child jobs to collect uniquely named process objects. Since it is a thread safe object, it can be safely used while the jobs run concurrently in the process.

$threadSafeDictionary = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
$jobs = Get-Process | ForEach {
    Start-ThreadJob {
        $proc = $using:_
        $dict = $using:threadSafeDictionary
        $dict.TryAdd($proc.ProcessName, $proc)
    }
}
$jobs | Wait-Job | Receive-Job

$threadSafeDictionary.Count
96

$threadSafeDictionary["pwsh"]

NPM(K)  PM(M)   WS(M) CPU(s)    Id SI ProcessName
------  -----   ----- ------    -- -- -----------
  112  108.25  124.43  69.75 16272  1 pwsh

See also