Azure Automation Accounts Az.Accounts PowerShell module broken?

Philipp Durrer 10

Since May 31st 2024 mid day UTC I see all our PowerShell 7.1 Runtime jobs failing due to the following error:

Failed
[91mWrite-Error: [0mC:\usr\src\tmp\accaf8e0-f99e-4d6b-8a7f-47b42ff95a77\runbooks\StartStop-AKS.ps1:223
[96mLine |
[96m 223 | [0m             [96mStopStartAksCluster[0m
[96m     | [91m             ~~~~~~~~~~~~~~~~~~~
[91m[96m     | [91mSystem.IO.FileLoadException: Could not load file or assembly
[96m     | [91m'Microsoft.ApplicationInsights, Version=2.18.0.315,
[96m     | [91mCulture=neutral, PublicKeyToken=31bf3856ad364e35'.    at
[96m     | [91mSystem.Management.Automation.Runspaces.InitialSessionState.Bind_LoadAssemblies(ExecutionContext context)    at System.Management.Automation.Runspaces.InitialSessionState.Bind(ExecutionContext context, Boolean updateOnly, PSModuleInfo module, Boolean noClobber, Boolean local, Boolean setLocation)    at Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(String moduleManifestPath, ExternalScriptInfo manifestScriptInfo, Hashtable data, Hashtable localizedData, ManifestProcessingFlags manifestProcessingFlags, Version minimumVersion, Version maximumVersion, Version requiredVersion, Nullable`1 requiredModuleGuid, ImportModuleOptions& options, Boolean& containedErrors)
[0m
[91mWrite-Error: [0mC:\usr\src\tmp\accaf8e0-f99e-4d6b-8a7f-47b42ff95a77\runbooks\StartStop-AKS.ps1:223
[96mLine |
[96m 223 | [0m             [96mStopStartAksCluster[0m
[96m     | [91m             ~~~~~~~~~~~~~~~~~~~
[91m[96m     | [91mSystem.IO.FileLoadException: Could not load file or assembly
[96m     | [91m'Microsoft.ApplicationInsights, Version=2.18.0.315,
[96m     | [91mCulture=neutral, PublicKeyToken=31bf3856ad364e35'.    at
[96m     | [91mSystem.Management.Automation.Runspaces.InitialSessionState.Bind_LoadAssemblies(ExecutionContext context)    at System.Management.Automation.Runspaces.InitialSessionState.Bind(ExecutionContext context, Boolean updateOnly, PSModuleInfo module, Boolean noClobber, Boolean local, Boolean setLocation)    at Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(String moduleManifestPath, ExternalScriptInfo manifestScriptInfo, Hashtable data, Hashtable localizedData, ManifestProcessingFlags manifestProcessingFlags, Version minimumVersion, Version maximumVersion, Version requiredVersion, Nullable`1 requiredModuleGuid, ImportModuleOptions& options, Boolean& containedErrors)
[0m
[91mOperationStopped: [0mC:\usr\src\tmp\accaf8e0-f99e-4d6b-8a7f-47b42ff95a77\runbooks\StartStop-AKS.ps1:149
[96mLine |
[96m 149 | [0m         [96mthrow $_.Exception[0m
[96m     | [91m         ~~~~~~~~~~~~~~~~~~
[91m[96m     | [91mCould not load file or assembly
[96m     | [91m'Microsoft.ApplicationInsights, Version=2.18.0.315,
[96m     | [91mCulture=neutral, PublicKeyToken=31bf3856ad364e35'.
[0m

My script is basically trying to do the following:

        #Inizialiting connection to the AutomationAccount
        try
        {
            Import-Module Az.Accounts
            "Logging in to Azure using Managed Identity..."
            Connect-AzAccount -Identity
        }

It looks like it doesn't matter the PowerShell version nor if I use a custom Runtime Environment or not everything fails. It's as if the Automation Account execution environment changed architecture and still tries to import the x64 modules.

Jesus Chao 141 Reputation points

2024-06-05T12:57:12.08+00:00
Hi Phillip,

Is this running on a Hybrid Worker or on the Azure Sandbox? When using the Runtime Environment, did you create a custom environment and which version of the AZ modules did you select? I have 11.2.0 of the AZ modules selected. Seems to be working ok.

Based on seeing the ANSI codes in your output, I assume that this is running in Hybrid but I could be wrong. If you are running on Hybrid, make sure you are installing the AZ powershell modules using a Powershell 7 cmd prompt on the Hybrid server. This will import the modules into "c:\program files\powershell\modules" and the Powershell session will then use those modules exclusively instead of possibly using the Windows Powershell (5.1) modules.

Also, if you want to get rid of the pesky ANSI code you are seeing in your output strings, add this to the top of your runbook script:

$PSStyle.OutputRendering = [System.Management.Automation.OutputRendering]::PlainText

Hope this helps
Philipp Durrer 10 Reputation points

2024-06-05T13:03:41.3533333+00:00
We're not using any Hybrid Workers as we don't want to use any more VMs. The test pane says it's using the "Run on Azure" mode, whatever that means.

I've tested the System Runtimes:

Powershell 5.1

Powershell 7.1

Powershell 7.2

They all fail with the same error. I've then created a custom runtime where I've explicitly added Az.Accounts as module from PSGallery however that fails in the same way as the system runtimes. As said something MUST have changed on Azure side on May 31st that these jobs are now all failing.

Can you maybe try to reproduce this in a test setup of your own and let me know if it works?

We use the system-managed managed-identity for the Connect-AzAccount step.

Jesus Chao 141

I am not a microsoft employee or anything so I dont have the means to reproduce exactly what you have. Just a normal user trying to help out....

In any case, I do have a custom 7.2 environment currently with the AZ 11.2.0 modules loaded running on the Azure sandbox and I am not seeing this issue. Based on the error, I guess its failing when it goes to import the module. Looks like the version it is trying to import is 2.18.0.

This is what I attempted...seems to work ok for me. My connect to Azure code is a little different from yours but I think it does the same thing.

import-module az.accounts -verbose:$false

Try {Disable-AzContextAutosave -Scope Process -erroraction stop | out-null}
catch {Throw "Error clearing previous Azure Context.  Error: $($_.exception.message)"}

#Connect to Azure with system-assigned managed identity
$attempt = 0
Do {
    Try {$AzureContext1 = Connect-AzAccount -Identity -erroraction stop -WarningAction Ignore -InformationAction Ignore}
    catch 
    {
        $attempt++
        If ($attempt -gt 6){Throw "Error connecting to Azure after $($attempt) attempts.  Error: $($_.exception.message)"}
        else {Start-Sleep -s 30}
    }
}until ($AzureContext1)

#Set Context and add to a global variable
Try {$global:AzureContext = Set-AzContext -SubscriptionName $AzureContext1.context.Subscription -DefaultProfile $AzureContext1.context -erroraction stop}
catch {Throw "Error setting new Context.  Error: $($_.exception.message)"}

#Output New Azure Context
$AzureContext

get-module

Disconnect-azaccount | out-null

On the Get-Module, the output shows that the Azure Sandbox is using Az.Accounts version 3.0

User's image

Here is a screen shot of our custom environment

User's image

Philipp Durrer 10

Isn't 2.18.0 referring to the ApplicationInsights module?

I've now setup a HybridWorker VM (ubuntu 20.04 based) and see slightly different errors:

{
[31;1mImport-Module: [0m/home/hweautomation/run/sandboxes/2e9bfc28-a84b-4f6c-8ef4-6140c158f91b/StartStop-AKS3a0b0cb7-3abb-4303-a439-878600beca82.ps1:37
"cluster": "narnia",
[36;1mLine |
"regions": [
[36;1m  37 | [0m             [36;1mImport-Module Az.Accounts[0m
"de"
[36;1m     | [31;1m             ~~~~~~~~~~~~~~~~~~~~~~~~~
]
[31;1m[36;1m     | [31;1mThe specified module 'Az.Accounts' was not loaded because no valid
}
[36;1m     | [31;1mmodule file was found in any module directory.
===========================================================================
[0m
Will stop aks-de-narnia-pool on rg-de-aks-narnia-pool
[31;1mWrite-Error: [0m/home/hweautomation/run/sandboxes/2e9bfc28-a84b-4f6c-8ef4-6140c158f91b/StartStop-AKS3a0b0cb7-3abb-4303-a439-878600beca82.ps1:225
===========================================================================
[36;1mLine |
Logging in to Azure using Managed Identity...
[36;1m 225 | [0m             [36;1mStopStartAksCluster[0m
[36;1m     | [31;1m             ~~~~~~~~~~~~~~~~~~~
[31;1m[36;1m     | [31;1mSystem.Management.Automation.CommandNotFoundException: The term
[36;1m     | [31;1m'Connect-AzAccount' is not recognized as a name of a cmdlet, function,
[36;1m     | [31;1mscript file, or executable program. Check the spelling of the name, or
[36;1m     | [31;1mif a path was included, verify that the path is correct and try again.
[36;1m     | [31;1mat
[36;1m     | [31;1mSystem.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)    at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
[0m
[31;1mWrite-Error: [0m/home/hweautomation/run/sandboxes/2e9bfc28-a84b-4f6c-8ef4-6140c158f91b/StartStop-AKS3a0b0cb7-3abb-4303-a439-878600beca82.ps1:225
[36;1mLine |
[36;1m 225 | [0m             [36;1mStopStartAksCluster[0m
[36;1m     | [31;1m             ~~~~~~~~~~~~~~~~~~~
[31;1m[36;1m     | [31;1mSystem.Management.Automation.CommandNotFoundException: The term
[36;1m     | [31;1m'Connect-AzAccount' is not recognized as a name of a cmdlet, function,
[36;1m     | [31;1mscript file, or executable program. Check the spelling of the name, or
[36;1m     | [31;1mif a path was included, verify that the path is correct and try again.
[36;1m     | [31;1mat
[36;1m     | [31;1mSystem.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)    at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
[0m
[31;1mConnect-AzAccount: [0m/home/hweautomation/run/sandboxes/2e9bfc28-a84b-4f6c-8ef4-6140c158f91b/StartStop-AKS3a0b0cb7-3abb-4303-a439-878600beca82.ps1:39
[36;1mLine |
[36;1m  39 | [0m             [36;1mConnect-AzAccount[0m -Identity
[36;1m     | [31;1m             ~~~~~~~~~~~~~~~~~
[31;1m[36;1m     | [31;1mThe term 'Connect-AzAccount' is not recognized as a name of a cmdlet,
[36;1m     | [31;1mfunction, script file, or executable program. Check the spelling of the
[36;1m     | [31;1mname, or if a path was included, verify that the path is correct and try
[36;1m     | [31;1magain.
[0m

That's the output I get. inthe test pane regardless of your output plane lines...

$PSStyle.OutputRendering = [System.Management.Automation.OutputRendering]::PlainText

I've installed Powershell 7.4 on that VM and the module using Install-Module Az -Scope AllUsers, did give me the same errors... then downgraded to pwsh core 7.2 which also didn't help. I really don't wanna pay for a Windows license just for a few start/stop scripts...

Jesus Chao 141 Reputation points

2024-06-05T14:29:18.86+00:00

Yeah - dont bother with the Hybrid stuff. That will just confuse things. I was just wondering if you were using that or not. Let's stick to the Azure Sandbox (Run on Azure).

What is interesting is that you are getting the ANSI codes when running your environment on the Azure sandbox. Not sure if I have seen that yet - but then again I am also not seeing the errors you are seeing in the first place.

Can you create just a new Powershell 7.2 environment from scratch and make sure you select the Az 11.2.0 modules during creation? When you go through the wizard, I believe it gives you the opportunity to select what AZ and Azure CLI versions you want.

Once you have that created, try the test again with a new runbook that just has the code:

Import-Module Az.Accounts

Get-Module

See if you get the same errors.
Philipp Durrer 10 Reputation points

2024-06-05T14:47:49.3733333+00:00

Hmmm that does indeed work. Now if Azure Portal was only mobile phone friendly I could copy past my old script to the new run book and see if that works.
Philipp Durrer 10 Reputation points

2024-06-06T07:56:58.4466667+00:00

Ok... WTF when I copy my script from the old Runbook to the new one it works. But the old runbook still throws the same error... god how I hate MS products...
Jesus Chao 141 Reputation points

2024-06-06T12:59:00.1833333+00:00

Well at least its working! And if Microsoft products were so good, we would be out of a job I guess :D

Don't boot me moderators
Philipp Durrer 10 Reputation points

2024-06-06T13:16:49.3666667+00:00

We'll see tomorrow. I've replaced all the runbooks and updated the schedules now. Thanks for the help Jesus
GP-3323 0 Reputation points

2024-06-10T20:25:51.1633333+00:00

We had the same issue. However, with the same code, it only occurred in the runbook that was running in the Powershell 7.1 environment, whereas the Powershell 5.1 environment runbook is still running fine.

I followed the steps here to update the runtime environment of the runbook to 7.2, and that appears to have resolved it for us.