We've been running into an issue lately with a specific piece of software with their release for 2024 (ProSeries 2024 if anyone is familiar). We're set up with a server/client installation and not standalone. I've involved the vendor repeatedly since the issue started to happen back in December but they just wash their hands on it.
Details:
- We have years 2017 - 2024 installed. All versions except the newest one work without any issues at all.
- Vendor software has been uninstalled and reinstalled several times. I've even personally uninstalled it and manually removed the registry entries, appdata folders, and everything related to that year in program files and programdata.
- All workstations are current on Windows 10. (We've budgeted and approved replacements and will be upgrading to Windows 11, but that's months away.)
- All workstations are on most recent updates and have nothing further to push.
- All firmware, BIOS, drivers from the SI have been installed and are current and up to date. SI is Lenovo.
- .NET is 4.8.1 / 533325 / 4.8.0.09037 (I believe this is the most recent for Windows 10, but correct me if I am mistaken -- at least the updater/installer detects it's on the latest one).
- .NET repair tool has also been run on machines. Came back clean every time.
- Vendor has worked with and re-installed everything on the admin/RDS server where the server component resides. We've verified that it runs on that (Server 2022 Standard) without any issues. They've signed off that it's installed via their recommended setups.
- Verified that there are no sleep settings on any of the machines, and verified in Device manager any power saving/sleep settings on the actual network card is turned off.
- We're on a flat network (no VLANs), small userbase. 6 people total, so it's not a large or complex environment.
- Any client machine attempting to run the workstation variant after installing immediately blue screens. This is every single workstation, reproduceable.
- BSOD happens in normal Windows environment and in safe mode.
- SI diagnostics were run on the machines and all came back clean.
BSOD Dump Below (through WinDBG):
************* Preparing the environment for Debugger Extensions Gallery repositories **************
ExtensionRepository : Implicit
UseExperimentalFeatureForNugetShare : true
AllowNugetExeUpdate : true
NonInteractiveNuget : true
AllowNugetMSCredentialProviderInstall : true
AllowParallelInitializationOfLocalRepositories : true
EnableRedirectToChakraJsProvider : false
-- Configuring repositories
----> Repository : LocalInstalled, Enabled: true
----> Repository : UserExtensions, Enabled: true
>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds
************* Waiting for Debugger Extensions Gallery to Initialize **************
>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.016 seconds
----> Repository : UserExtensions, Enabled: true, Packages count: 0
----> Repository : LocalInstalled, Enabled: true, Packages count: 42
Microsoft (R) Windows Debugger Version 10.0.27725.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\NOPENOPENOPE\Downloads\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 19041 MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Kernel base = 0xfffff8074b800000 PsLoadedModuleList = 0xfffff807
4c42a3e0
Debug session time: Tue Jan 7 11:17:50.053 2025 (UTC - 5:00)
System Uptime: 0 days 0:21:44.986
Loading Kernel Symbols
………………………………………………………
…….Page 506672 not present in the dump file. Type ".hh dbgerr004" for details
…………………………………………………
……………………………………………..
Loading User Symbols
PEB is paged out (Peb.Ldr = 00000096`ddee1018). Type ".hh dbgerr001" for details
Loading unloaded module list
…….
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff8074bbfe8f0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffc301
335776e0=0000000000000103
2: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
MUP_FILE_SYSTEM (103)
MUP file system detected an error.
Arguments:
Arg1: 0000000000000001, MUP_BUGCHECK_NO_FILECONTEXT
Could not locate MUP file context corresponding to a file object.
Arg2: ffffe08a875b0c20, IRP Address if an IRP was used, NULL otherwise.
Arg3: ffffe08a8fe8b0d0, FILE_OBJECT Address whose MUP file context could not be found
Arg4: ffffe08a79d60b00, DEVICE_OBJECT Address
Debugging Details:
*** WARNING: Check Image - Checksum mismatch - Dump: 0x2504b, File: 0x2d3f1 - C:\ProgramData\Dbg\sym\mup.sys\47232DD926000\mup.sys
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 2000
Key : Analysis.Elapsed.mSec
Value: 3160
Key : Analysis.IO.Other.Mb
Value: 11
Key : Analysis.IO.Read.Mb
Value: 1
Key : Analysis.IO.Write.Mb
Value: 22
Key : Analysis.Init.CPU.mSec
Value: 937
Key : Analysis.Init.Elapsed.mSec
Value: 8512
Key : Analysis.Memory.CommitPeak.Mb
Value: 92
Key : Analysis.Version.DbgEng
Value: 10.0.27725.1000
Key : Analysis.Version.Description
Value: 10.2408.27.01 amd64fre
Key : Analysis.Version.Ext
Value: 1.2408.27.1
Key : Bugcheck.Code.KiBugCheckData
Value: 0x103
Key : Bugcheck.Code.LegacyAPI
Value: 0x103
Key : Bugcheck.Code.TargetModel
Value: 0x103
Key : Failure.Bucket
Value: 0x103\_mup!MupSurrogateRegisterProvider
Key : Failure.Hash
Value: {3e1653ee-3ca3-05ee-6b7e-64d59c83bf4b}
Key : Hypervisor.Enlightenments.Value
Value: 0
Key : Hypervisor.Enlightenments.ValueHex
Value: 0
Key : Hypervisor.Flags.AnyHypervisorPresent
Value: 0
Key : Hypervisor.Flags.ApicEnlightened
Value: 0
Key : Hypervisor.Flags.ApicVirtualizationAvailable
Value: 0
Key : Hypervisor.Flags.AsyncMemoryHint
Value: 0
Key : Hypervisor.Flags.CoreSchedulerRequested
Value: 0
Key : Hypervisor.Flags.CpuManager
Value: 0
Key : Hypervisor.Flags.DeprecateAutoEoi
Value: 0
Key : Hypervisor.Flags.DynamicCpuDisabled
Value: 0
Key : Hypervisor.Flags.Epf
Value: 0
Key : Hypervisor.Flags.ExtendedProcessorMasks
Value: 0
Key : Hypervisor.Flags.HardwareMbecAvailable
Value: 1
Key : Hypervisor.Flags.MaxBankNumber
Value: 0
Key : Hypervisor.Flags.MemoryZeroingControl
Value: 0
Key : Hypervisor.Flags.NoExtendedRangeFlush
Value: 0
Key : Hypervisor.Flags.NoNonArchCoreSharing
Value: 0
Key : Hypervisor.Flags.Phase0InitDone
Value: 0
Key : Hypervisor.Flags.PowerSchedulerQos
Value: 0
Key : Hypervisor.Flags.RootScheduler
Value: 0
Key : Hypervisor.Flags.SynicAvailable
Value: 0
Key : Hypervisor.Flags.UseQpcBias
Value: 0
Key : Hypervisor.Flags.Value
Value: 131072
Key : Hypervisor.Flags.ValueHex
Value: 20000
Key : Hypervisor.Flags.VpAssistPage
Value: 0
Key : Hypervisor.Flags.VsmAvailable
Value: 0
Key : Hypervisor.RootFlags.AccessStats
Value: 0
Key : Hypervisor.RootFlags.CrashdumpEnlightened
Value: 0
Key : Hypervisor.RootFlags.CreateVirtualProcessor
Value: 0
Key : Hypervisor.RootFlags.DisableHyperthreading
Value: 0
Key : Hypervisor.RootFlags.HostTimelineSync
Value: 0
Key : Hypervisor.RootFlags.HypervisorDebuggingEnabled
Value: 0
Key : Hypervisor.RootFlags.IsHyperV
Value: 0
Key : Hypervisor.RootFlags.LivedumpEnlightened
Value: 0
Key : Hypervisor.RootFlags.MapDeviceInterrupt
Value: 0
Key : Hypervisor.RootFlags.MceEnlightened
Value: 0
Key : Hypervisor.RootFlags.Nested
Value: 0
Key : Hypervisor.RootFlags.StartLogicalProcessor
Value: 0
Key : Hypervisor.RootFlags.Value
Value: 0
Key : Hypervisor.RootFlags.ValueHex
Value: 0
Key : SecureKernel.HalpHvciEnabled
Value: 0
Key : WER.OS.Branch
Value: vb\_release
Key : WER.OS.Version
Value: 10.0.19041.1
BUGCHECK_CODE: 103
BUGCHECK_P1: 1
BUGCHECK_P2: ffffe08a875b0c20
BUGCHECK_P3: ffffe08a8fe8b0d0
BUGCHECK_P4: ffffe08a79d60b00
FILE_IN_CAB: MEMORY.DMP
FAULTING_THREAD: ffffe08a849f4080
IRP_ADDRESS: ffffe08a875b0c20
DEVICE_OBJECT: ffffe08a79d60b00
DRIVER_OBJECT: ffffe08a79d608f0
IMAGE_NAME: mup.sys
MODULE_NAME: mup
FAULTING_MODULE: fffff80752840000 mup
BLACKBOXBSD: 1 (!blackboxbsd)
BLACKBOXNTFS: 1 (!blackboxntfs)
BLACKBOXWINLOGON: 1
PROCESS_NAME: protax24.exe
STACK_TEXT:
ffffc301335776d8 fffff807
52855601 : 0000000000000103 00000000
00000001 ffffe08a875b0c20 ffffe08a
8fe8b0d0 : nt!KeBugCheckEx
ffffc301335776e0 fffff807
4ba4a295 : ffffe08a8fe8b0d0 00000000
00000000 0000000000000000 00000000
00000001 : mup!MupSurrogateRegisterProvider+0x7f1
ffffc30133577750 fffff807
4b134a76 : 0000000000000001 00000000
20206f49 00000096df9ff3b0 00000000
00000000 : nt!IofCallDriver+0x55
ffffc30133577790 fffff807
4ba4a295 : 0000000000000002 00000000
00000001 ffffe08a875b0c40 00000000
20206f49 : FLTMGR!FltpDispatch+0xd6
ffffc301335777f0 fffff807
4be3260c : 0000000000000000 ffffe08a
875b0c20 ffffe08a8fe8b0d0 00000096
00000000 : nt!IofCallDriver+0x55
ffffc30133577830 fffff807
4bdfb9a9 : ffffe08a00000000 ffffc301
33577a80 0000000000000000 ffffe08a
875b0c20 : nt!IopSynchronousServiceTail+0x34c
ffffc301335778d0 fffff807
4bc12905 : ffffe08a849f4080 00000096
df9ff398 0000000000000000 00000096
00000008 : nt!NtQueryVolumeInformationFile+0x4c9
ffffc30133577990 00007ff8
5a74de14 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : nt!KiSystemServiceCopyEnd+0x25
00000096df9ff378 00000000
00000000 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : 0x00007ff8`5a74de14
SYMBOL_NAME: mup!MupSurrogateRegisterProvider+7f1
STACK_COMMAND: .process /r /p 0xffffe08a8770b340; .thread 0xffffe08a849f4080 ; kb
BUCKET_ID_FUNC_OFFSET: 7f1
FAILURE_BUCKET_ID: 0x103_mup!MupSurrogateRegisterProvider
OS_VERSION: 10.0.19041.1
BUILDLAB_STR: vb_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {3e1653ee-3ca3-05ee-6b7e-64d59c83bf4b}
Followup: MachineOwner
If anyone has any recommendations or insight I'd gladly welcome it. As of this point I've been bashing my head against this for over two months and any outside comments/suggestions can only help.
*** Moved from Windows / Windows 10 / Performance and system failures ***