Creation of module with CLSID "{6620A03E-9E85-4BEA-947C-D508CDB540FE}" failed with error "%1 is not a valid Win32 application." in rule "discovery<SOME_NUMBER>.<SOME_GUID>" running for instance "<SOME_NAME>" with id:"{<SOME_GUID>}" in management group "<SOME_MG_NAME>".

1. we either are trying to load a 32bit binary file from a 64bit process (or the other way around - very uncommon ... but not impossible)
2. or there is something directly wrong with the binary file contents (some corruption at the file structure level / header): https://en.wikipedia.org/wiki/Portable_Executable

Add-Type @"
using System;
using System.Runtime.InteropServices;
public static class NativeMethods {
   [DllImport(@"$($env:ProgramFiles)\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll")]
   public static extern void DllCanUnloadNow();
}
"@
[NativeMethods]::DllCanUnloadNow()

• it will create a C# file which will get compiled in a temporary directory that will do the actual native (C/C++) DLL loading/import
• after this, we will call a standard DLL function (usually included in every DLL - verified that we have it in NetworkDiscoveryModules.dll because I can look at the code :p - but you can usually rely on the fact that almost all DLLs have this) - DllCanUnloadNow()
• this function - DllCanUnloadNow() - will not return anything from this PowerShell script on a computer where there is no issue with the DLL - however, it will return an error if there is something wrong with it - just like this one below which we got when running the script on the affected server:

1. we want to load DLL whatever.dll
2. we go verify if the binary file has the correct bitness and if it's not corrupt in any way
3. if all goes well, we check the import section and for each DLL which it is dependent on, we do the exact same thing one by one - steps 1 to 3 for each of them
4. if any of the DLLs in the import section cannot be correctly loaded (not correct bitness, corrupt in some way, etc.) we will fail and throw the error describing why the DLL cannot be loaded - however, we throw the error for the initial DLL we were trying to load, we don't specify if there actually was a failure in the loading of one of the dependent DLLs

1. kernelbase!LoadLibraryExW which is used to load DLLs by Windows OS (Ex comes from "Extended" which means being able to pass in more "options" - parameters and W comes from the fact that the UNICODE string version of it is being used) => we are just starting here in order to continue with  the investigation with other breakpoints only when we reach the point where we are trying to load the DLL we are interested in troubleshooting (in this case NetworkDiscoveryModules.dll)
2. ntdll!LdrpFindOrMapDll (not documented on MSDN, should not directly be used) which will get called by the DLL loader component (LDR) to try to find the DLL in different places, verify binary file structures/header, load all the DLL that it is dependent on (verifying consistency of each as well) and in the end load the actual DLL we call in order to be able to execute from it => this will be important because this is where we can get the DLL file name that we are currently trying to load
3. ntdll!ZwCreateSection which goes into Kernel and does the actual file read from disk and checks the required information (is it execution code, does it have a valid "MZ" start, is the header ok?, does it have the correct bitness?, etc.) => this is the actual function call we are interested in the end, because this is the one that will return the actual error in case that we have an issue loading the binary file to execute from it

0:000> bp kernelbase!LoadLibraryExW

0:000> g
Breakpoint 0 hit
kernelbase!LoadLibraryExW:
00007ffe`36b48d10 48895c2410 mov qword ptr [rsp+10h],rbx ss:0000000c`301d4b38=0000000c301d91a8

0:001> k
Child-SP RetAddr Call Site
0000000c`301d4b28 00007ffe`310ae4b4 kernelbase!LoadLibraryExW
0000000c`301d4b30 00007ffe`310ae421 clr!CLRLoadLibraryExWorker+0x54
0000000c`301d4b80 00007ffe`31098aff clr!CLRLoadLibraryEx+0x51
0000000c`301d4bd0 00007ffe`31098a28 clr!LoadedImageLayout::LoadedImageLayout+0xca
0000000c`301d4e90 00007ffe`313f6a84 clr!PEImageLayout::Load+0x3d
0000000c`301d4ed0 00007ffe`313f69de clr!PEImage::GetLayoutInternal+0x361930
0000000c`301d4f20 00007ffe`3107ee8f clr!PEImage::GetLayout+0x36195e
0000000c`301d4fa0 00007ffe`3107ed53 clr!RuntimeOpenImageInternal+0xcf
0000000c`301d5080 00007ffe`3107ec7b clr!GetAssemblyMDInternalImportEx+0xcd
0000000c`301d5100 00007ffe`310d3cc9 clr!CreateMetaDataImport+0x1b
0000000c`301d5140 00007ffe`310d3c1d clr!BindResult::Init+0x69
0000000c`301d51b0 00007ffe`3118867b clr!BindResult::CreateFromPath+0xc9
0000000c`301d5220 00007ffe`31188891 clr!NATIVE_BINDER_SPACE::VerifyNativeCandidate+0x55f
0000000c`301d58b0 00007ffe`31086b40 clr!NATIVE_BINDER_SPACE::CreateVerifiedNICandidate+0x10a
0000000c`301d5bc0 00007ffe`31085f39 clr!NATIVE_BINDER_SPACE::BindToNativeDependency+0x2ac
0000000c`301d5de0 00007ffe`31085803 clr!BindToNativeAssembly+0xed
0000000c`301d5f00 00007ffe`3108799f clr!BindToNativeImage+0x1bb
0000000c`301d6230 00007ffe`31082a6b clr!CAsmDownloadMgr::RecordInfoAndProbeNativeImage+0x106
0000000c`301d62a0 00007ffe`3108303b clr!CAsmDownloadMgr::PreDownloadCheck+0x518
0000000c`301d6ae0 00007ffe`31082e45 clr!CAssemblyDownload::PreDownload+0xb7
0000000c`301d6b80 00007ffe`3108251a clr!CAssemblyName::BindToObject+0x419
0000000c`301d6cd0 00007ffe`310822c2 clr!FusionBind::RemoteLoad+0x1aa
0000000c`301d6e10 00007ffe`31082047 clr!AssemblySpec::LoadAssembly+0x19a
0000000c`301d6f30 00007ffe`31081cf5 clr!AssemblySpec::FindAssemblyFile+0x113
0000000c`301d76d0 00007ffe`3108a4ae clr!AppDomain::BindAssemblySpec+0xef7
0000000c`301d8950 00007ffe`310ab18b clr!AssemblySpec::LoadDomainAssembly+0x1ec
0000000c`301d8f00 00007ffe`3118c9e8 clr!AssemblySpec::LoadAssembly+0x1b
0000000c`301d8f40 00007ffe`2f6ebde2 clr!AssemblyNative::Load+0x304
0000000c`301d92b0 00007ffe`304c4326 mscorlib_ni!System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Reflection.RuntimeAssembly, System.Threading.StackCrawlMark ByRef, IntPtr, Boolean, Boolean, Boolean)+0xd2
0000000c`301d9360 00007ffe`31209f51 mscorlib_ni!System.Reflection.RuntimeAssembly.LoadWithPartialNameInternal(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef)+0xd66ad6
0000000c`301d93e0 00007ffe`31209e4f clr!ExceptionTracker::CallHandler+0xc5
0000000c`301d9480 00007ffe`312081c7 clr!ExceptionTracker::CallCatchHandler+0x7f
0000000c`301d9510 00007ffe`396c347d clr!ProcessCLRException+0x2e6
0000000c`301d95f0 00007ffe`39685405 ntdll!RtlpExecuteHandlerForUnwind+0xd
0000000c`301d9620 00007ffe`31208260 ntdll!RtlUnwindEx+0x385
0000000c`301d9d00 00007ffe`3120821c clr!ClrUnwindEx+0x40
0000000c`301da220 00007ffe`396c33fd clr!ProcessCLRException+0x2b2
0000000c`301da300 00007ffe`39684847 ntdll!RtlpExecuteHandlerForException+0xd
0000000c`301da330 00007ffe`39683a6d ntdll!RtlDispatchException+0x197
0000000c`301daa00 00007ffe`36b495fc ntdll!RtlRaiseException+0x18d
0000000c`301db1c0 00007ffe`31209864 kernelbase!RaiseException+0x68
0000000c`301db2a0 00007ffe`314bb7ef clr!RaiseTheExceptionInternalOnly+0x2fe
0000000c`301db3a0 00007ffe`3118cb37 clr!UnwindAndContinueRethrowHelperAfterCatch+0x80
0000000c`301db3f0 00007ffe`2f75d99e clr!AssemblyNative::Load+0x453
0000000c`301db760 00007ffe`2ff13666 mscorlib_ni!System.Reflection.RuntimeAssembly.LoadWithPartialNameInternal(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef)+0x14e
0000000c`301db890 00007ffe`1933970f mscorlib_ni!System.Reflection.Assembly.LoadWithPartialName(System.String)+0x46
0000000c`301db8e0 00007ffe`193394e1 system_management_automation_ni!System.Management.Automation.ExecutionContext.LoadAssembly(System.String, System.String, System.Exception ByRef)+0x15f
0000000c`301db9a0 00007ffe`193211f2 system_management_automation_ni!System.Management.Automation.ExecutionContext.AddAssembly(System.String, System.String, System.Exception ByRef)+0x21
0000000c`301db9f0 00007ffe`19316887 system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadBinaryModule(System.Management.Automation.PSModuleInfo, Boolean, System.String, System.String, System.Reflection.Assembly, System.String, System.Management.Automation.SessionState, ImportModuleOptions, ManifestProcessingFlags, System.String, Boolean, Boolean, Boolean ByRef, System.String, Boolean)+0x452
0000000c`301dbbf0 00007ffe`1931b82b system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleNamedInManifest(System.Management.Automation.PSModuleInfo, Microsoft.PowerShell.Commands.ModuleSpecification, System.String, Boolean, System.String, System.Management.Automation.SessionState, ImportModuleOptions, ManifestProcessingFlags, Boolean, Boolean, System.Object, Boolean ByRef, System.String)+0x977
0000000c`301dbe20 00007ffe`19316ab5 system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(System.String, System.Management.Automation.ExternalScriptInfo, System.Collections.Hashtable, System.Collections.Hashtable, ManifestProcessingFlags, System.Version, System.Version, ImportModuleOptions ByRef, Boolean ByRef)+0x4d0b
0000000c`301dc9d0 00007ffe`1931fa1b system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(System.Management.Automation.ExternalScriptInfo, ManifestProcessingFlags, System.Version, System.Version, ImportModuleOptions ByRef)+0xd5
0000000c`301dca70 00007ffe`1932386c system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModule(System.Management.Automation.PSModuleInfo, System.String, System.String, System.String, System.Management.Automation.SessionState, System.Object, ImportModuleOptions ByRef, ManifestProcessingFlags, Boolean ByRef, Boolean ByRef)+0xc4b
0000000c`301dcda0 00007ffe`193241d8 system_management_automation_ni!Microsoft.PowerShell.Commands.ImportModuleCommand.ImportModule_LocallyViaName(ImportModuleOptions, System.String)+0x4ac
0000000c`301dcef0 00007ffe`193dbb0e system_management_automation_ni!Microsoft.PowerShell.Commands.ImportModuleCommand.ProcessRecord()+0x198
0000000c`301dcfd0 00007ffe`19382b55 system_management_automation_ni!System.Management.Automation.CommandProcessor.ProcessRecord()+0x29e
0000000c`301dd080 00007ffe`1937953e system_management_automation_ni!System.Management.Automation.CommandProcessorBase.DoExecute()+0xe5
0000000c`301dd0f0 00007ffe`193666d0 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object, System.Collections.Hashtable, Boolean)+0x17e
0000000c`301dd1e0 00007ffe`1936581f system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()+0x750
0000000c`301dd400 00007ffe`19365241 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()+0x2af
0000000c`301dd550 00007ffe`193348d5 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.StartPipelineExecution()+0x321
0000000c`301dd5d0 00007ffe`19363fe4 system_management_automation_ni!System.Management.Automation.Runspaces.PipelineBase.CoreInvoke(System.Collections.IEnumerable, Boolean)+0x305
0000000c`301dd650 00007ffe`193df48e system_management_automation_ni!System.Management.Automation.Runspaces.PipelineBase.Invoke(System.Collections.IEnumerable)+0x24
0000000c`301dd690 00007ffe`193deff0 system_management_automation_ni!System.Management.Automation.PowerShell+Worker.ConstructPipelineAndDoWork(System.Management.Automation.Runspaces.Runspace, Boolean)+0x2de
0000000c`301dd770 00007ffe`193d9fbe system_management_automation_ni!System.Management.Automation.PowerShell.CoreInvokeHelper[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSInvocationSettings)+0x350
0000000c`301dd820 00007ffe`193d9aca system_management_automation_ni!System.Management.Automation.PowerShell.CoreInvoke[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSInvocationSettings)+0x4de
0000000c`301dd9b0 00007ffe`193efca0 system_management_automation_ni!System.Management.Automation.PowerShell.CoreInvoke[[System.__Canon, mscorlib]](System.Collections.IEnumerable, System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSInvocationSettings)+0x1ba
0000000c`301dda60 00007ffe`193ed6c2 system_management_automation_ni!System.Management.Automation.CommandDiscovery.AutoloadSpecifiedModule(System.String, System.Management.Automation.ExecutionContext, System.Management.Automation.SessionStateEntryVisibility, System.Exception ByRef)+0x2c0
0000000c`301ddb00 00007ffe`19376326 system_management_automation_ni!System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(System.String, System.Management.Automation.ExecutionContext, System.String, System.Management.Automation.CommandOrigin, System.Management.Automation.SearchResolutionOptions, System.Management.Automation.CommandTypes, System.Exception ByRef)+0x312
0000000c`301ddc10 00007ffe`19375ed1 system_management_automation_ni!System.Management.Automation.CommandDiscovery.LookupCommandInfo(System.String, System.Management.Automation.CommandTypes, System.Management.Automation.SearchResolutionOptions, System.Management.Automation.CommandOrigin, System.Management.Automation.ExecutionContext)+0x3d6
0000000c`301ddcf0 00007ffe`193dd8ce system_management_automation_ni!System.Management.Automation.CommandDiscovery.LookupCommandProcessor(System.String, System.Management.Automation.CommandOrigin, System.Nullable`1<Boolean>)+0x31
0000000c`301ddd50 00007ffe`193dcfd1 system_management_automation_ni!System.Management.Automation.ExecutionContext.CreateCommand(System.String, Boolean)+0x6e
0000000c`301dddb0 00007ffe`19335255 system_management_automation_ni!System.Management.Automation.PipelineOps.AddCommand(System.Management.Automation.Internal.PipelineProcessor, System.Management.Automation.CommandParameterInternal[], System.Management.Automation.Language.CommandBaseAst, System.Management.Automation.CommandRedirection[], System.Management.Automation.ExecutionContext)+0x3f1
0000000c`301ddef0 00007ffe`1a0a036e system_management_automation_ni!System.Management.Automation.PipelineOps.InvokePipeline(System.Object, Boolean, System.Management.Automation.CommandParameterInternal[][], System.Management.Automation.Language.CommandBaseAst[], System.Management.Automation.CommandRedirection[][], System.Management.Automation.Language.FunctionContext)+0x135
0000000c`301ddfb0 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.ActionCallInstruction`6[[System.__Canon, mscorlib],[System.Boolean, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x19e
0000000c`301de050 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de160 00007ffe`1938575b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de270 00007ffe`193752af system_management_automation_ni!System.Management.Automation.Interpreter.Interpreter.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x7b
0000000c`301de2f0 00007ffe`19383925 system_management_automation_ni!System.Management.Automation.Interpreter.LightLambda.RunVoid1[[System.__Canon, mscorlib]](System.__Canon)+0x11f
0000000c`301de380 00007ffe`193832f8 system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.RunClause(System.Action`1<System.Management.Automation.Language.FunctionContext>, System.Object, System.Object)+0x405
0000000c`301de510 00007ffe`19382fce system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.Complete()+0x1a8
0000000c`301de5c0 00007ffe`19382d01 system_management_automation_ni!System.Management.Automation.CommandProcessorBase.DoComplete()+0x16e
0000000c`301de680 00007ffe`19379551 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(System.Management.Automation.CommandProcessorBase)+0xb1
0000000c`301de720 00007ffe`193666d0 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object, System.Collections.Hashtable, Boolean)+0x191
0000000c`301de810 00007ffe`1936581f system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()+0x750
0000000c`301dea30 00007ffe`192e7001 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()+0x2af
0000000c`301deb80 00007ffe`2f6d2d45 system_management_automation_ni!System.Management.Automation.Runspaces.PipelineThread.WorkerProc()+0x31
0000000c`301debb0 00007ffe`2f6d2ab9 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x285
0000000c`301ded10 00007ffe`2f6d2a97 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x9
0000000c`301ded40 00007ffe`2f6ea161 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x57
0000000c`301ded90 00007ffe`3103afb3 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x51
0000000c`301dede0 00007ffe`3103ae9e clr!CallDescrWorkerInternal+0x83
0000000c`301dee20 00007ffe`3103b632 clr!CallDescrWorkerWithHandler+0x4a
0000000c`301dee60 00007ffe`31146bd9 clr!MethodDescCallSite::CallTargetWorker+0x251
0000000c`301df010 00007ffe`3103c8e1 clr!ThreadNative::KickOffThread_Worker+0x105
0000000c`301df270 00007ffe`3103c868 clr!ManagedThreadBase_DispatchInner+0x2d
0000000c`301df2b0 00007ffe`3103c7d9 clr!ManagedThreadBase_DispatchMiddle+0x6c
0000000c`301df3b0 00007ffe`3103c91b clr!ManagedThreadBase_DispatchOuter+0x75
0000000c`301df440 00007ffe`31146aba clr!ManagedThreadBase_FullTransitionWithAD+0x2f
0000000c`301df4a0 00007ffe`31153656 clr!ThreadNative::KickOffThread+0xd2
0000000c`301df570 00007ffe`38f113d2 clr!Thread::intermediateThreadProc+0x7d
0000000c`301df7b0 00007ffe`396454e4 kernel32!BaseThreadInitThunk+0x22
0000000c`301df7e0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

0:001> du @rcx
0000000c`2f4dd240 "C:\Windows\assembly\NativeImages"
0000000c`2f4dd280 "_v4.0.30319_64\Microsoft.P521220"
0000000c`2f4dd2c0 "ea#\ab32ec62ecc4b90efc07602a8353"
0000000c`2f4dd300 "8c65\Microsoft.PowerShell.Comman"
0000000c`2f4dd340 "ds.Utility.ni.dll"

0:001> g
Breakpoint 0 hit
kernelbase!LoadLibraryExW:
00007ffe`36b48d10 48895c2410 mov qword ptr [rsp+10h],rbx ss:0000000c`301dcbd8=0000000c301dddf8
0:001> du @rcx
0000000c`301dd078 "C:\Program Files\Microsoft Syste"
0000000c`301dd0b8 "m Center 2012 R2\Operations Mana"
0000000c`301dd0f8 "ger\Server\NetworkDiscoveryModul"
0000000c`301dd138 "es.dll"

0:001> bp ntdll!LdrpFindOrMapDll
0:001> bp ntdll!ZwCreateSection
0:001> g
Breakpoint 2 hit
ntdll!LdrpFindOrMapDll:
00007ffe`396a61c0 4053            push    rbx

0:001> dx ((ntdll!_UNICODE_STRING *)@rcx)
((ntdll!_UNICODE_STRING *)0x00000000c301dc948) : 0xc301dc948 : "C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll" [Type: _UNICODE_STRING *]
[<Raw View>] [Type: _UNICODE_STRING]

0:001> g
Breakpoint 1 hit
ntdll!NtCreateSection:
00007ffe`396c0b50 4c8bd1 mov r10,rcx

0:001> k
Child-SP RetAddr Call Site
0000000c`301dc538 00007ffe`396a6921 ntdll!NtCreateSection
0000000c`301dc540 00007ffe`3965b2b5 ntdll!LdrpFindOrMapDll+0x761
0000000c`301dc8b0 00007ffe`39698d69 ntdll!LdrpLoadDll+0x295
0000000c`301dcae0 00007ffe`36b48dda ntdll!LdrLoadDll+0x99
0000000c`301dcb60 00007ffe`310ae4b4 kernelbase!LoadLibraryExW+0xca
0000000c`301dcbd0 00007ffe`310ae421 clr!CLRLoadLibraryExWorker+0x54
0000000c`301dcc20 00007ffe`31197dfd clr!CLRLoadLibraryEx+0x51
0000000c`301dcc70 00007ffe`3140bf1e clr!LocalLoadLibraryHelper+0x31
0000000c`301dcca0 00007ffe`310af2a4 clr!NDirect::LoadLibraryModule+0x35c7de
0000000c`301dd4f0 00007ffe`310af50a clr!NDirect::NDirectLink+0x80
0000000c`301dd820 00007ffe`310af469 clr!NDirect::GetStubForILStub+0x4a
0000000c`301dd870 00007ffe`310af54b clr!GetStubForInteropMethod+0x65
0000000c`301dd8b0 00007ffe`310a9696 clr!MethodDesc::DoPrestub+0xca7
0000000c`301dda80 00007ffe`3103226a clr!PreStubWorker+0x3d6
0000000c`301ddd90 00007ffd`d1a40a1c clr!ThePreStub+0x5a
0000000c`301dde60 00007ffe`2cb64c1c DynamicClass.CallSite.Target(System.Runtime.CompilerServices.Closure, System.Runtime.CompilerServices.CallSite, System.Type)+0xfc
0000000c`301ddee0 00007ffe`1938b088 system_core_ni!System.Dynamic.UpdateDelegates.UpdateAndExecute1[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Runtime.CompilerServices.CallSite, System.__Canon)+0x39c
0000000c`301ddff0 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.DynamicInstruction`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x58
0000000c`301de050 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de160 00007ffe`1938575b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de270 00007ffe`193752af system_management_automation_ni!System.Management.Automation.Interpreter.Interpreter.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x7b
0000000c`301de2f0 00007ffe`19383925 system_management_automation_ni!System.Management.Automation.Interpreter.LightLambda.RunVoid1[[System.__Canon, mscorlib]](System.__Canon)+0x11f
0000000c`301de380 00007ffe`193832f8 system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.RunClause(System.Action`1<System.Management.Automation.Language.FunctionContext>, System.Object, System.Object)+0x405
0000000c`301de510 00007ffe`19382fce system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.Complete()+0x1a8
0000000c`301de5c0 00007ffe`19382d01 system_management_automation_ni!System.Management.Automation.CommandProcessorBase.DoComplete()+0x16e
0000000c`301de680 00007ffe`19379551 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(System.Management.Automation.CommandProcessorBase)+0xb1
0000000c`301de720 00007ffe`193666d0 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object, System.Collections.Hashtable, Boolean)+0x191
0000000c`301de810 00007ffe`1936581f system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()+0x750
0000000c`301dea30 00007ffe`192e7001 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()+0x2af
0000000c`301deb80 00007ffe`2f6d2d45 system_management_automation_ni!System.Management.Automation.Runspaces.PipelineThread.WorkerProc()+0x31
0000000c`301debb0 00007ffe`2f6d2ab9 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x285
0000000c`301ded10 00007ffe`2f6d2a97 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x9
0000000c`301ded40 00007ffe`2f6ea161 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x57
0000000c`301ded90 00007ffe`3103afb3 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x51
0000000c`301dede0 00007ffe`3103ae9e clr!CallDescrWorkerInternal+0x83
0000000c`301dee20 00007ffe`3103b632 clr!CallDescrWorkerWithHandler+0x4a
0000000c`301dee60 00007ffe`31146bd9 clr!MethodDescCallSite::CallTargetWorker+0x251
0000000c`301df010 00007ffe`3103c8e1 clr!ThreadNative::KickOffThread_Worker+0x105
0000000c`301df270 00007ffe`3103c868 clr!ManagedThreadBase_DispatchInner+0x2d
0000000c`301df2b0 00007ffe`3103c7d9 clr!ManagedThreadBase_DispatchMiddle+0x6c
0000000c`301df3b0 00007ffe`3103c91b clr!ManagedThreadBase_DispatchOuter+0x75
0000000c`301df440 00007ffe`31146aba clr!ManagedThreadBase_FullTransitionWithAD+0x2f
0000000c`301df4a0 00007ffe`31153656 clr!ThreadNative::KickOffThread+0xd2
0000000c`301df570 00007ffe`38f113d2 clr!Thread::intermediateThreadProc+0x7d
0000000c`301df7b0 00007ffe`396454e4 kernel32!BaseThreadInitThunk+0x22
0000000c`301df7e0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

0:001> pt
ntdll!NtCreateSection+0xa:
00007ffe`396c0b5a c3 ret

0:001> r @rax
rax=0000000000000000

0:001> g
Breakpoint 2 hit
ntdll!LdrpFindOrMapDll:
00007ffe`396a61c0 4053 push rbx

0:001> dx ((ntdll!_UNICODE_STRING *)@rcx)
((ntdll!_UNICODE_STRING *)@rcx) : 0xc301dc4b0 : "dmboot.dll" [Type: _UNICODE_STRING *]
[<Raw View>] [Type: _UNICODE_STRING]

0:001> g
Breakpoint 1 hit
ntdll!NtCreateSection:
00007ffe`396c0b50 4c8bd1 mov r10,rcx

0:001> pt
ntdll!NtCreateSection+0xa:
00007ffe`396c0b5a c3 ret

0:001> r @rax
rax=00000000c000012f

0:001> !error 00000000c000012f
Error code: (NTSTATUS) 0xc000012f (3221225775) - The specified image file did not have the correct format, it did not have an initial MZ.

• some filter driver (like anti-virus software or anything else like "my-super-cool-awesome-disk-speed-up.sys" and so on ...
• computer power-failure in the middle of specific type of disk IO (direct access) - very unlikely though ...
• network (possibly filter) driver while copying a file from some other computer (or some network device in between that somehow ends up modifying bits)
• super cool awesome storage controller that directly manipulates (RAM) memory through the mother board (not through OS Kernel)
• some totally other random weird issue like a CPU bit-flip while the file was being written to disk ?! ...

"use strict";

class __DllLoadStatus
{
   get Ldr()
   {
      return new __DllLoadFailure(this);
   }
}

class __DllLoadFailure
{
   constructor(process)
   {
      this.__process = process;
   }

   Initialize()
   {
      var ctl = host.namespace.Debugger.Utility.Control;
      ctl.ExecuteCommand("bp /1 ntdll!ZwCreateSection;g");
      ctl.ExecuteCommand("pt");

      var address = host.currentThread.Registers.User.rip;
      ctl.ExecuteCommand("bp ntdll!LdrpFindOrMapDll \"dx @$curprocess.Ldr.GetDllName();gc\"");
      ctl.ExecuteCommand("bp " + address.toString(16) + " \"dx @$curprocess.Ldr.GetLoadStatus();gc\"");

      host.diagnostics.debugLog("Ok, we are all set! Run \"g\" to start automated debugging.\n");
   }

   GetDllName()
   {
      var ctl = host.namespace.Debugger.Utility.Control;
      var address = host.currentThread.Registers.User.rcx;

      var pDllName = host.createPointerObject(address, "ntdll.dll", "_UNICODE_STRING *");
      var dllName = host.memory.readWideString(pDllName.dereference().Buffer).toString();

      host.diagnostics.debugLog(dllName, "\n");
   }

   GetLoadStatus()
   {
      var ctl = host.namespace.Debugger.Utility.Control;
      var status = ctl.ExecuteCommand("!error @rax").First().toString();

      host.diagnostics.debugLog(status, "\n");
   }
}

function initializeScript()
{
   return [
      new host.namedModelParent(__DllLoadStatus, "Debugger.Models.Process")
   ];
}

0:004> !load jsprovider
0:004> .scriptload C:\DbgScripts\LdrExtensionTest.js
JavaScript script successfully loaded from 'C:\DbgScripts\LdrExtensionTest.js'

0:000> dx @$curprocess.Ldr.Initialize()
Ok, we are all set! Run "g" to start automated debugging.
@$curprocess.Ldr.Initialize()
 
0:001> g
C:\Windows\system32\rsaenh.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCR100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
KERNEL32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
USER32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ole32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
OLEAUT32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ADVAPI32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-server.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-discovery.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-auto-discovery.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCP100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ntdll.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
RPCRT4.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
SHLWAPI.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
HealthServiceRuntime.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
KERNEL32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCP100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCR100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
KERNEL32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ADVAPI32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-clsapi.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
dmboot.dll
@$curprocess.Ldr.GetDllName()
Error code: (NTSTATUS) 0xc000012f (3221225775) - The specified image file did not have the correct format, it did not have an initial MZ.
@$curprocess.Ldr.GetLoadStatus()

5: kd> !mex.mem
Page File: \??\C:\pagefile.sys
Current: 10794940 Kb Free Space: 6592416 Kb ( 10.29 GB)
Minimum: 10794940 Kb Maximum: 10794940 Kb ( 10.29 GB)

Physical Memory: 8385237 ( 33540948 Kb) ( 31.99 GB)
Available Pages: 27177 ( 108708 Kb) ( 106.16 MB)
ResAvail Pages: 8193973 ( 32775892 Kb) ( 31.26 GB)
Locked IO Pages: 0 ( 0 Kb) ( 0)
Free System PTEs: 33514422 ( 134057688 Kb) ( 127.85 GB)
Modified Pages: 4073135 ( 16292540 Kb) ( 15.54 GB)
Modified PF Pages: 3 ( 12 Kb) ( 12.00 KB)
Modified No Write Pages: 0 ( 0 Kb) ( 0)
NonPagedPool 0: 8326 ( 33304 Kb) ( 32.52 MB)
NonPagedPool 1: 6844 ( 27376 Kb) ( 26.73 MB)
NonPagedPool 2: 0 ( 0 Kb) ( 0)
NonPagedPool 3: 0 ( 0 Kb) ( 0)
NonPagedPool Usage: 36694 ( 146776 Kb) ( 143.34 MB) Current pool size: 981.94 MB
NonPagedPool Max: 6263793 ( 25055172 Kb) ( 23.89 GB)
PagedPool 0: 107157 ( 428628 Kb) ( 418.58 MB)
PagedPool 1: 12274 ( 49096 Kb) ( 47.95 MB)
PagedPool 2: 6081 ( 24324 Kb) ( 23.75 MB)
PagedPool 3: 0 ( 0 Kb) ( 0)
PagedPool 4: 38 ( 152 Kb) ( 152.00 KB)
PagedPool Usage: 125550 ( 502200 Kb) ( 490.43 MB) WorkingSet: 286.58 MB, PeakWorkingSet: 517.27 MB
PagedPool Maximum: 33554432 ( 134217728 Kb) ( 128.00 GB)
Processor Commit: 3751 ( 15004 Kb) ( 14.65 MB)
Session Commit: 12901 ( 51604 Kb) ( 50.39 MB)
Syspart SharedCommit 0
Shared Commit: 17255 ( 69020 Kb) ( 67.40 MB)
Special Pool: 0 ( 0 Kb) ( 0)
Kernel Stacks: 13455 ( 53820 Kb) ( 52.56 MB)
Pages For MDLs: 1859 ( 7436 Kb) ( 7.26 MB)
Pages For AWE: 0 ( 0 Kb) ( 0)
NonPagedPool Commit: 0 ( 0 Kb) ( 0)
PagedPool Commit: 125678 ( 502712 Kb) ( 490.93 MB)
Driver Commit: 4240 ( 16960 Kb) ( 16.56 MB)
Boot Commit: 0 ( 0 Kb) ( 0)
System PageTables: 0 ( 0 Kb) ( 0)
VAD/PageTable Bitmaps: 3882 ( 15528 Kb) ( 15.16 MB)
ProcessLockedFilePages: 0 ( 0 Kb) ( 0)
Pagefile Hash Pages: 0 ( 0 Kb) ( 0)
Sum System Commit: 183021 ( 732084 Kb) ( 714.93 MB)
Total Private: 5012200 ( 20048800 Kb) ( 19.12 GB)
Misc/Transient Commit: 170655 ( 682620 Kb) ( 666.62 MB)
Committed pages: 5365876 ( 21463504 Kb) ( 20.47 GB)
Commit limit: 11083507 ( 44334028 Kb) ( 42.28 GB)

*** Low Available Pages (Standby+Zeroed+Free). This can cause performance issues. ***

Virtual Memory Physical Memory File Cache Cache Writes

• Available Pages: 27177 ( 108708 Kb)   ( 106.16 MB)           => so we only have 106.16 MB free physical memory (pages) ...
• Modified Pages: 4073135 ( 16292540 Kb)   ( 15.54 GB)      => from 32 GB total physical memory, we have 15.54 GB (half of the memory) in Modified Pages

5: kd> !mex.tl system
PID Address Name
=========== ================ =========================================
0x4 0n4 fffffa8019a6b040 System
0x1b8 0n440 fffffa8028f02060 svchost.exe(LocalSystemNetworkRestricted)
=========== ================ =========================================
PID Address Name

Warning! Zombie process(es) detected (not displayed). Count: 2 [zombie report]

5: kd> !mex.p fffffa8019a6b040
Name Address Ses PID User Name Create Time Up Time Mods Handle Act Thrd Z Thrd Parent
====== ================ ==== ======= =============== ========================== ============ ==== ====== ======== ====== ==============
System fffffa8019a6b040 none 4 (0n4) DOMAIN\COMPUTERNAME$ 06/26/2017 12:11:57.995 PM 2h:03:22.004 0 1176 204 0 Idle 0 (0n0)

Memory Details:

VM Peak Work Set Commit Size
======== ======== ======== ===========
23.18 MB 27.99 MB 18.49 MB 556 KB

CPU Details:

User Kernel Total
==== ========== ==========
0s 24m:33.696 24m:33.696

Show Threads: Unique Stacks !mex.listthreads (!lt) fffffa8019a6b040 !winde.lp fffffa8019a6b040 !process fffffa8019a6b040 7

5: kd> !mex.us nt!MiModifiedPageWriter
1 thread [stats]: fffffa8019ab43c0
fffff80001eda6aa nt!KiSwapContext+0x7a
fffff80001edd142 nt!KiCommitThreadWait+0x1d2
fffff80001e9c03b nt!KeWaitForGate+0xfb
fffff80001e73c3a nt!MiModifiedPageWriter+0x5a
fffff80002170f12 nt!PspSystemThreadStartup+0x5a
fffff80001ec9de6 nt!KiStartSystemThread+0x16

Threads matching filter: 1 out of 204

5: kd> !mex.t fffffa8019ab43c0
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time
System (fffffa8019a6b040) fffffa8019ab43c0 4.ac 0s 1s.154 3135 WrFreePage 3s.650

WaitBlockList:
Object Name Type Other Waiters
fffff80002092e00 nt!MmModifiedPageWriterGate+0x0 Gate 0

Priority:
Current Base UB FB IO Page
17 8 0 0 2 5

# Child-SP Return Call Site
0 fffff880029fdac0 fffff80001edd142 nt!KiSwapContext+0x7a
1 fffff880029fdc00 fffff80001e9c03b nt!KiCommitThreadWait+0x1d2
2 fffff880029fdc90 fffff80001e73c3a nt!KeWaitForGate+0xfb
3 fffff880029fdce0 fffff80002170f12 nt!MiModifiedPageWriter+0x5a
4 fffff880029fdd40 fffff80001ec9de6 nt!PspSystemThreadStartup+0x5a
5 fffff880029fdd80 0000000000000000 nt!KiStartSystemThread+0x16

0: kd> dx -r1 ((nt!_KGATE *)0xfffff80002092e00)
((nt!_KGATE *)0xfffff80002092e00)                 : 0xfffff80002092e00 [Type: _KGATE *]
    [+0x000] Header           [Type: _DISPATCHER_HEADER]

0: kd> dx -r1 ((ntkrnlmp!_DISPATCHER_HEADER *)0xfffff80002092e00)
((ntkrnlmp!_DISPATCHER_HEADER *)0xfffff80002092e00)                 : 0xfffff80002092e00 [Type: _DISPATCHER_HEADER *]
    [+0x000] Type             : 0x7 [Type: unsigned char]
    [+0x001] TimerControlFlags : 0x1 [Type: unsigned char]
    [+0x001 ( 0: 0)] Absolute         : 0x1 [Type: unsigned char]
    [+0x001 ( 1: 1)] Coalescable      : 0x0 [Type: unsigned char]
    [+0x001 ( 2: 2)] KeepShifting     : 0x0 [Type: unsigned char]
    [+0x001 ( 7: 3)] EncodedTolerableDelay : 0x0 [Type: unsigned char]
    [+0x001] Abandoned        : 0x1 [Type: unsigned char]
    [+0x001] Signalling       : 0x1 [Type: unsigned char]
    [+0x002] ThreadControlFlags : 0x6 [Type: unsigned char]
    [+0x002 ( 0: 0)] CpuThrottled     : 0x0 [Type: unsigned char]
    [+0x002 ( 1: 1)] CycleProfiling   : 0x1 [Type: unsigned char]
    [+0x002 ( 2: 2)] CounterProfiling : 0x1 [Type: unsigned char]
    [+0x002 ( 7: 3)] Reserved         : 0x0 [Type: unsigned char]
    [+0x002] Hand             : 0x6 [Type: unsigned char]
    [+0x002] Size             : 0x6 [Type: unsigned char]
    [+0x003] TimerMiscFlags   : 0x0 [Type: unsigned char]
    [+0x003 ( 5: 0)] Index            : 0x0 [Type: unsigned char]
    [+0x003 ( 6: 6)] Inserted         : 0x0 [Type: unsigned char]
    [+0x003 ( 7: 7)] Expired          : 0x0 [Type: unsigned char]
    [+0x003] DebugActive      : 0x0 [Type: unsigned char]
    [+0x003 ( 0: 0)] ActiveDR7        : 0x0 [Type: unsigned char]
    [+0x003 ( 1: 1)] Instrumented     : 0x0 [Type: unsigned char]
    [+0x003 ( 5: 2)] Reserved2        : 0x0 [Type: unsigned char]
    [+0x003 ( 6: 6)] UmsScheduled     : 0x0 [Type: unsigned char]
    [+0x003 ( 7: 7)] UmsPrimary       : 0x0 [Type: unsigned char]
    [+0x003] DpcActive        : 0x0 [Type: unsigned char]
    [+0x000] Lock             : 393479 [Type: long]
    [+0x004] SignalState      : 0 [Type: long]
    [+0x008] WaitListHead     [Type: _LIST_ENTRY]

5: kd> !mex.us nt!MiMappedPageWriter
1 thread [stats]: fffffa8019ab5040
fffff80001eda6aa nt!KiSwapContext+0x7a
fffff80001edd142 nt!KiCommitThreadWait+0x1d2
fffff80001ede056 nt!KeDelayExecutionThread+0x186
fffff80001f117bc nt!MiGatherMappedPages+0x8c
fffff80001f122f8 nt!MiMappedPageWriter+0x198
fffff80002170f12 nt!PspSystemThreadStartup+0x5a
fffff80001ec9de6 nt!KiStartSystemThread+0x16

Threads matching filter: 1 out of 204
5: kd> !mex.t fffffa8019ab5040
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time
System (fffffa8019a6b040) fffffa8019ab5040 4.b0 0s 187ms 133350 DelayExecution 15ms

Priority:
Current Base UB FB IO Page
17 8 0 0 2 5

# Child-SP Return Call Site
0 fffff88002b08910 fffff80001edd142 nt!KiSwapContext+0x7a
1 fffff88002b08a50 fffff80001ede056 nt!KiCommitThreadWait+0x1d2
2 fffff88002b08ae0 fffff80001f117bc nt!KeDelayExecutionThread+0x186
3 fffff88002b08b50 fffff80001f122f8 nt!MiGatherMappedPages+0x8c
4 fffff88002b08c50 fffff80002170f12 nt!MiMappedPageWriter+0x198
5 fffff88002b08d40 fffff80001ec9de6 nt!PspSystemThreadStartup+0x5a
6 fffff88002b08d80 0000000000000000 nt!KiStartSystemThread+0x16

5: kd> !mex.mem -fc
Name ControlArea FsContext Valid StandbyDirty Shared Locked Total
=================================================================================== ================ ================ ======== ============ ====== ====== ========
[...SNIPPED...]
$LogFile fffffa801fc6b990 fffffa801f656e10 860 KB 860 KB
CM_123.mdf fffffa802a3dd490 fffff8a003a85140 1.75 MB 1.75 MB
$Mft fffffa801ce9d4a0 fffffa801ce7dbb0 2.82 MB 2.82 MB
CM_123.mdf fffffa80204ffd30 fffff8a003423140 10.69 MB 15.54 GB 15.55 GB

Virtual Memory Physical Memory File Cache Cache Writes

5: kd> !kdexts.ca fffffa80204ffd30
ControlArea @ fffffa80204ffd30
Segment fffff8a004a8b940 Flink 0000000000000000 Blink 0000000000000000
Section Ref 1 Pfn Ref 3e30f0 Mapped Views 19f
User Ref 0 WaitForDel 0 Flush Count 0
File Object fffffa802a4144f0 ModWriteCount 0 System Views 19f
WritableRefs 0
Flags (8080) File WasPurged

\SCCMSQLBackup\123Backup\SiteDBServer\CM_123.mdf

Segment @ fffff8a004a8b940
ControlArea fffffa80204ffd30 ExtendInfo 0000000000000000
Total Ptes 1745000
Segment Size 1745000000 Committed 0
Flags (c0000) ProtectionMask

Subsection 1 @ fffffa80204ffdb0
ControlArea fffffa80204ffd30 Starting Sector 0 Number Of Sectors 800000
Base Pte fffff8a014a00000 Ptes In Subsect 800000 Unused Ptes 0
Flags d Sector Offset 0 Protection 6
Accessed
Flink fffffa801fc069d0 Blink fffffa802a3dd560 MappedViews 0

Subsection 2 @ fffffa801fc0a010
ControlArea fffffa80204ffd30 Starting Sector 800000 Number Of Sectors 800000
Base Pte fffff8a01ca00000 Ptes In Subsect 800000 Unused Ptes 0
Flags d Sector Offset 0 Protection 6
Accessed
Flink 0000000000000000 Blink 0000000000000000 MappedViews 19f

Subsection 3 @ fffffa801fc54b30
ControlArea fffffa80204ffd30 Starting Sector 1000000 Number Of Sectors 745000
Base Pte 0000000000000000 Ptes In Subsect 745000 Unused Ptes 0
Flags c Sector Offset 0 Protection 6
Flink 0000000000000000 Blink 0000000000000000 MappedViews 0

5: kd> !kdexts.defwrites
*** Cache Write Throttle Analysis ***

CcTotalDirtyPages: 4075797 (16303188 Kb)
CcDirtyPageThreshold: 4058054 (16232216 Kb)
AvailablePages: 27177 ( 108708 Kb)
ThrottleTop: 450 ( 1800 Kb)
ThrottleBottom: 80 ( 320 Kb)
ModifiedPages: 4073135 (16292540 Kb)

CcTotalDirtyPages >= CcDirtyPageThreshold, writes throttled

Check these thread(s): CcWriteBehind(LazyWriter)
Check critical workqueue for the lazy writer, !exqueue 16
Cc Deferred Write list: (CcDeferredWrites)
File: fffffa802a4144f0 Event: fffff880074906d0
File: fffffa8029c08070 Event: fffff88007968640

5: kd> !kdexts.fileobj fffffa802a4144f0

\SCCMSQLBackup\123Backup\SiteDBServer\CM_123.mdf

Device Object: 0xfffffa801f7d07c0 \Driver\volmgr
Vpb: 0xfffffa8029683ad0
Event signalled
Access: Read Write Delete

Flags: 0x43062
Synchronous IO
Sequential Only
Cache Supported
Modified
Size Changed
Handle Created

File Object is currently busy and has 0 waiters.

FsContext: 0xfffff8a003423140 FsContext2: 0xfffff8a00f5d3500
Private Cache Map: 0xfffffa802029b9c0
CurrentByteOffset: edb0f0000
Cache Data:
Section Object Pointers: fffffa8029479818
Shared Cache Map: fffffa802029b850 File Offset: ffffffffdb0f0000

5: kd> !mex.findhandle fffffa802a4144f0
smssqlbkup.exe fffffa802935eb10
384

5: kd> !mex.p fffffa802935eb10
Name Address Ses PID User Name Create Time Up Time Mods Handle Act Thrd Z Thrd Parent
============== ================ === ============ =============== ========================== ============ ==== ====== ======== ====== ========================
smssqlbkup.exe fffffa802935eb10 0 614 (0n1556) DOMAIN\COMPUTERNAME$ 06/26/2017 12:13:00.723 PM 2h:02:19.276 54 221 8 0 services.exe 264 (0n612)

Command Line: "D:\SMS_FOLDER\bin\x64\smssqlbkup.exe"

Memory Details:

VM Peak Work Set Commit Size PP Quota NPP Quota
======== ========= ======== =========== ========= =========
86.53 MB 107.85 MB 66.95 MB 12.94 MB 137.89 KB 15.38 KB

CPU Details:

User Kernel Total
====== ========= =========
2s.403 2m:21.447 2m:23.850

Show Threads: Unique Stacks !mex.listthreads (!lt) fffffa802935eb10 !winde.lp fffffa802935eb10 !process fffffa802935eb10 7

Show LPC Port information for process

5: kd> !mex.lt
Process PID Thread Id Time Reason
============== === ================ ==== ======= ==============
smssqlbkup.exe 614 fffffa8029361b50 5c0 2s.371 UserRequest
smssqlbkup.exe 614 fffffa801f6a5b50 53c 43s.649 DelayExecution
smssqlbkup.exe 614 fffffa801f6a55c0 874 514ms Executive
smssqlbkup.exe 614 fffffa80293b2b50 c38 43s.649 UserRequest
smssqlbkup.exe 614 fffffa80295c0b50 7d4 43s.649 UserRequest
smssqlbkup.exe 614 fffffa801f929b50 11fc 43s.649 WrQueue
smssqlbkup.exe 614 fffffa8021324060 1af4 2s.386 WrQueue
smssqlbkup.exe 614 fffffa8029610060 10a0 2s.371 WrQueue

Thread Count: 8

5: kd> !mex.t fffffa801f6a55c0
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time COM-Initialized
smssqlbkup.exe (fffffa802935eb10) fffffa801f6a55c0 614.874 1s.841 2m:21.228 36298 Executive 514ms APTKIND_MULTITHREADED (MTA)

WaitBlockList:
Object Type Other Waiters
fffff880074906d0 NotificationEvent 0

# Child-SP Return Call Site
0 fffff88007490430 fffff80001edd142 nt!KiSwapContext+0x7a
1 fffff88007490570 fffff80001edf96f nt!KiCommitThreadWait+0x1d2
2 fffff88007490600 fffff80001ea2527 nt!KeWaitForSingleObject+0x19f
3 fffff880074906a0 fffff880016ebcc8 nt!CcCanIWrite+0xfffffffffffa11f7
4 fffff88007490770 fffff88001401102 Ntfs!NtfsCopyWriteA+0x68
5 fffff88007490970 fffff880014048ba fltmgr!FltpPerformFastIoCall+0xf2
6 fffff880074909d0 fffff8800142283e fltmgr!FltpPassThroughFastIo+0xda
7 fffff88007490a10 fffff800021f1bbe fltmgr!FltpFastIoWrite+0x1ce
8 fffff88007490ab0 fffff80001ed70d3 nt!NtWriteFile+0x5ad
9 fffff88007490bb0 0000000077a2bdba nt!KiSystemServiceCopyEnd+0x13
a 00000000014cce38 000007fefd73865c ntdll!NtWriteFile+0xa
b 00000000014cce40 00000000778d171a KERNELBASE!WriteFile+0xfe
c 00000000014cceb0 00000000778d1ea5 kernel32!BaseCopyStream+0x4b2
d 00000000014cd8a0 00000000778d1907 kernel32!BasepCopyFileExW+0x545
e 00000000014cdda0 00000000779556f2 kernel32!CopyFileExW+0x97
f 00000000014cde20 000000013ff6fb4a kernel32!CopyFileA+0x62
10 00000000014cde80 000000013ff81a5c smssqlbkup+0x9fb4a
11 00000000014cdf10 000000013ff7c430 smssqlbkup+0xb1a5c
12 00000000014ce280 000000013ff7bb5a smssqlbkup+0xac430
13 00000000014cf3b0 000000013ff7753f smssqlbkup+0xabb5a
14 00000000014cf400 000000013ff7987c smssqlbkup+0xa753f
15 00000000014cf6f0 000000013ff85643 smssqlbkup+0xa987c
16 00000000014cf7c0 0000000140155d55 smssqlbkup+0xb5643
17 00000000014cf9e0 0000000140155bec smssqlbkup+0x285d55
18 00000000014cfa10 00000000778d59cd smssqlbkup+0x285bec
19 00000000014cfa40 0000000077a0a561 kernel32!BaseThreadInitThunk+0xd
1a 00000000014cfa70 0000000000000000 ntdll!RtlUserThreadStart+0x1d

• first of all we are clearly trying to write to a file (nt!NtWriteFile after the usermode call to CopyFile because the smssqlbkup.exe process is trying to copy the file)
• second of all we are waiting in nt!CcCanIWrite (remember explanation of throttling from above)

5: kd> !mex.ioq
Collecting Data...This might take a moment.
Collecting Raid Data...Returned Failure
Collecting Disk Data...Added Object Disk with Filter DISK
Returned Success

================== Disk.sys Devices ==========================

Device Name DeviceObject DeviceExtension PrivateFdoData PendingIRPs
===================== ================ ================ ================ ===========
\Device\Harddisk0\DR0 fffffa801cd28060 fffffa801cd281b0 fffffa801cd29010 2
==================RaidPort Info: \Device\RaidPort2 ==========================
Device Name Miniport DeviceExtension BusType IoModel
=================================== ======== ================ =========== =========================
\Device\RaidPort2(fffffa801a668060) somestoragedriver fffffa801a6681b0 BusTypeRAID StorSynchronizeFullDuplex
RaidUnit(LUN) ClassPNPDevice DeviceState QueueState OutstandingCount QueueDepth PendingIrps
================ ================ ================== ========== ================ ========== ===========
fffffa801a6b2930 fffffa801cd28060 DeviceStateWorking Normal 2 254 2

Summary Notes:
IOQ found 80 devices, of which 2 have a total of 4 pending Irps.

5: kd> !mex.ioq -e fffffa801cd281b0

============================================
Disk Information: \Device\Harddisk0\DR0
============================================
ClassPNP Devices(Disk.sys) Information:
============================================
Device Name DeviceObject DeviceExtension PrivateFdoData PendingIRPs
===================== ================ ================ ================ ===========
\Device\Harddisk0\DR0 fffffa801cd28060 fffffa801cd281b0 fffffa801cd29010 2

Transfer Packet Current Irp Thread Wait Thread #Retries TimeOut (s) DeviceName CurrentDrvr Original Irp FileName Srb SrbStatus
================ ================ =========== ============================== ======== =========== =============== =========== ================ ============= ================ ==================
fffffa801d11a310 fffffa801d11b010 15ms fffffa80213cc430(RAMMap64.exe) 0 0 HarddiskVolume2 somestoragedriver fffffa8029b27010 \pagefile.sys fffffa801d11a430 SRB_STATUS_PENDING
fffffa801d1fdb10 fffffa801d1fd910 0s fffffa8029573b50(sqlservr.exe) 0 0 HarddiskVolume2 somestoragedriver fffffa801f8ab980 \pagefile.sys fffffa801d1fdc30 SRB_STATUS_PENDING

5: kd> !mex.mirp -v fffffa801d11b010

Irp Details: fffffa801d11b010 [ verbose | !ddt | !winde.io | !irp ]

Mdl : fffffa801fd8a7b0
System buffer :
Issuing Process :
Thread :
Frame Count : 2
IoStatus Status : c00000bb
IoStatus Info : 0000000000000000
Requester Mode :
Cancel : 0
Cancel IRQL : 0
Apc Environment : 0
User Iosb : 0000000000000000
User Event :
APC : 0000000000000000
Completion Key : 0000000000000000
Cancel Routine :
Original File Object: 0000000000000000
Original File Name :

Irp Stack Frame(s)

# Driver Major Minor Dispatch Routine Control Code Flg Ctrl Status Completion Invoker(s) Device QueueLocation File Context Completion Routine Args
=== ================ ======================= ===== ======================== ============ === ==== ======= ====================== ================ ============= ====== ====================== ============================ ===============================================================================
1 CREATE 0 0 0 None 0000000000000000 (null) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
->2 \Driver\somestoragedriver INTERNAL_DEVICE_CONTROL 0 storport!RaDriverScsiIrp 12 e1 Pending Cancel, Success, Error fffffa801a6b27e0 (null) fffffa801d11a310(CPnp) CLASSPNP!TransferPktComplete fffffa801d11a430(CPnp) 0000000000000000 0000000000000000 fffffa801a6b2930(Devi)

IO Status: 0xc00000bb (The request is not supported.)

 

5: kd> !mex.mirp -v fffffa801d1fd910

Irp Details: fffffa801d1fd910 [ verbose | !ddt | !winde.io | !irp ]

Mdl : fffffa801f5b9670
System buffer :
Issuing Process :
Thread :
Frame Count : 2
IoStatus Status : c00000bb
IoStatus Info : 0000000000000000
Requester Mode :
Cancel : 0
Cancel IRQL : 0
Apc Environment : 0
User Iosb : 0000000000000000
User Event :
APC : 0000000000000000
Completion Key : 0000000000000000
Cancel Routine :
Original File Object: 0000000000000000
Original File Name :

Irp Stack Frame(s)

# Driver Major Minor Dispatch Routine Control Code Flg Ctrl Status Completion Invoker(s) Device QueueLocation File Context Completion Routine Args
=== ================ ======================= ===== ======================== ============ === ==== ======= ====================== ================ ============= ====== ====================== ============================ ===============================================================================
1 CREATE 0 0 0 None 0000000000000000 (null) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
->2 \Driver\somestoragedriver INTERNAL_DEVICE_CONTROL 0 storport!RaDriverScsiIrp 12 e1 Pending Cancel, Success, Error fffffa801a6b27e0 (null) fffffa801d1fdb10(CPnp) CLASSPNP!TransferPktComplete fffffa801d1fdc30(CPnp) 0000000000000000 0000000000000000 fffffa801a6b2930(Devi)

IO Status: 0xc00000bb (The request is not supported.)

5: kd> lmtm somestoragedriver
Browse full module list
start end module name
fffff880`01549000 fffff880`01559000 somestoragedriver Mon Aug 9 21:21:24 2010 (4C604724)

• Update the "somestoragedriver.sys" driver to the latest existing version
• After this, monitor disk performance using Performance Monitor and check if the disk is performing well

5: kd> !mex.mem -p
Loading PFN database... Building kernel map...this will take a while.

Top physical RAM users:

Type Name ControlArea/PID Valid Standby Dirty Shared Locked PageTables TOTAL
=========== ============== ================ ========= ======== ======== ======= ========= ========== =========
Mapped File CM_123.mdf FFFFFA80204FFD30 10.69 MB 15.54 GB 15.55 GB
Process sqlservr.exe 14866272 14.18 GB 79.96 MB 16 KB n/a n/a 35.34 MB 14.29 GB
Process ReportingServi FFFFFA80292A6B10 241.43 MB 2.62 MB 16 KB n/a n/a 2.28 MB 246.35 MB
System NonPaged Pool 184.33 MB n/a 184.33 MB 184.33 MB
Process RAMMap64.exe FFFFFA8020C87380 111.77 MB 4.94 MB n/a n/a 2.06 MB 118.77 MB
Process some_other_process FFFFFA8029411060 97.23 MB 1.47 MB n/a n/a 1.18 MB 99.88 MB
System Kernel Stacks 53.94 MB 44 KB n/a 992 KB 128 KB 54.11 MB
Process procexp64.exe FFFFFA80297EB060 36.13 MB n/a n/a 608 KB 36.72 MB
Process svchost.exe FFFFFA8028EF25D0 28.81 MB 1.97 MB n/a n/a 732 KB 31.49 MB
Process explorer.exe FFFFFA8029673A40 23.09 MB 196 KB n/a n/a 724 KB 23.99 MB
Process perfmon.exe FFFFFA802976A6C0 21.61 MB 160 KB n/a n/a 460 KB 22.21 MB
Mapped File sqlservr.exe FFFFFA8029838AC0 20.22 MB 20.22 MB
Process explorer.exe FFFFFA80296AAB10 19.06 MB 120 KB n/a n/a 664 KB 19.82 MB
Process lsass.exe FFFFFA801F106A70 18.47 MB 96 KB n/a n/a 352 KB 18.91 MB
Process WmiPrvSE.exe FFFFFA80294AF060 18.11 MB 4 KB n/a n/a 548 KB 18.64 MB
Process smsexec.exe FFFFFA801F5D0B10 17.64 MB 40 KB n/a n/a 724 KB 18.38 MB
Process procexp64.exe FFFFFA802A574B10 17.18 MB 40 KB n/a n/a 460 KB 17.66 MB
Mapped File shell32.dll FFFFFA801EF80680 13.5 MB 3.12 MB 16.62 MB
Process svchost.exe FFFFFA8028F0BB10 15.03 MB 28 KB n/a n/a 432 KB 15.48 MB
Process some_process FFFFFA8029018B10 14.19 MB 196 KB n/a n/a 432 KB 14.8 MB
=========== ============== ================ ========= ======== ======== ======= ========= ========== =========
Type Name ControlArea/PID Valid Standby Dirty Shared Locked PageTables TOTAL

Only the top 20 of 1612 rows are being shown. Show all

Virtual Memory Physical Memory File Cache Cache Writes

5: kd> !kdexts.memusage
[...SNIPPED...]
Usage Summary (in Kb):
Control Valid Standby Dirty Shared Locked PageTables name
ffffffffffffd 8384 0 0 0 0 0 AWE
fffffa8019a34010 440 0 0 184 0 0 mapped_file( winhttp.dll )
fffffa8019ac18d0 16 0 0 0 0 0 mapped_file( netutils.dll )
[...SNIPPED...]
fffffa80204ffd30 10944 0 16292096 0 0 0 mapped_file( CM_123.mdf )
[...SNIPPED...]

1. Create a new Service Request using the Default Service Request Template (uses the default/standard Template for each WorkItem type)
2. Add the Title & Description of the Incident to the Service Request
3. Add the common WorkItem relationships to the Service Request & delete these relationships from the Incident (less data in DB, better performance)
4. Add the Incident to the Service Request as related WorkItem (WorkItemRelatesToWorkItem)
5. Create a new AnalystCommentLog and add it to the Incident as a related AnalystCommentLog containing a comment that the Incident was closed by this transform Task and also containing the new Service Request ID
6. Close the Incident (set the Status to Closed)

• close Incidents and add comment on closing (setting Status to Closed, creating AnalystCommentLog and adding it to the Incident as related AnalystCommentLog)
• creating and editing the WorkItem of the specific Task being executed (ex. if the "Transform to SR" is being called for example, then Create/Edit permissions are needed for Service Requests)

<ConsoleTask ID="TransformIncidentToServiceRequest" Accessibility="Public" Enabled="true" Target="Incident!System.WorkItem.Incident" RequireOutput="false">
  <Assembly>Console!SdkDataAccessAssembly</Assembly>
  <Handler>Microsoft.EnterpriseManagement.UI.SdkDataAccess.ConsoleTaskHandler</Handler>
  <Parameters>
    <Argument Name="Assembly">CustomTransformWorkItemTasks</Argument>
    <Argument Name="Type">CustomTransformWorkItemTasks.TransformTaskHandler</Argument>
    <Argument>Service</Argument>
  </Parameters>
</ConsoleTask>

1. Assembly (needed named argument that allows the Handler to know what DLL it needs to use for this): CustomTransformWorkItemTasks
2. Type (needed named argument that tells the Handler which is the class from our DLL that is handling this): CustomTransformWorkItemTasks.TransformTaskHandler
3. A 3rd unnamed argument, which is actually the 1st argument that we are passing to our method (CustomTransformWorkItemTasks.TransformTaskHandler) which is a simple string for which we are manually checking if it is set in our method: Service

• note that the (main) namespace is called CustomTransformWorkItemTasks like our DLL will be called as well
• also notice, that we define a class in this namespace called TransformTaskHandler that extends the (SM build-in Microsoft.EnterpriseManagement.UI.SdkDataAccess.ConsoleCommand class)

namespace CustomTransformWorkItemTasks
{
    public class TransformTaskHandler : ConsoleCommand
    {

public override void ExecuteCommand(IList nodes, NavigationModelNodeTask task, ICollection parameters)
{

if (parameters.Contains("Service"))
{
    // do stuff here - the actual work you need to be done in your task when this argument is passed by the Task
    // in this case, I am setting some variables that decide what I will create later in the actual processing based on the WorkItem type (check the entire source code on GitHub)
}

• notice that we are using EnterpriseManagementObjectProjection with Class Type constructor in order to be able to create Class & Relationship objects together
• we are just adding the Relationships we create to the class object, by adding it to the EnterpriseManagementObjectProjection class using the Add(...) Method
• in order to "really" save/create the Class & Relationship objects of this projection into the CMDB, which we are currently only having in memory, we now need to call Overwrite() Method on the projection
• when we create the AnalystCommentLog, we are using CreatableEnterpriseManagementObject because it's an object that does not exist yet and we are just creating it, but we don't need to use a projection (EnterpriseManagementObjectProjection) because we will not create & add any Relationship to it (we are just adding the newly object itself as a related object to the new Service Request)
• we need to make sure to set the correct Status for the new WorkItem in order for the internal workflows to correctly process it in order for it to get in progress (Status will be New or Active depending on the WorkItem type)
• it is also very important to set the Prefixes for each Activity (depending on the type) that exists in the WorkItem Template we apply (which in this case is the default Template for each WorkItem type)
• another special case we need to worry about, is when we are cloning the Relationships - if the RelationshipType is derived from System.Membership, it is very important to create a brand new Target object copied from the existing one (new BaseManagedEntityId) because Relationships of this type (Membership) have a Target object which is bound to it's source (it cannot have a Relationship to 2 different objects)
• all the MP, Class, etc. names are defined in a separate CS file called Constants when different classes are defined - check the actual source code on GitHub to understand those
• check the comments for more detailed explanations

// here is the code that does the actual work
// we wrap this around a try/catch block to handle any exception that may happen and display it in case of failure

try
{
    // getting the types we need based on the string variables we have filled earlier based on the WorkItem type we need to create

    ManagementPack workItemMp = emg.GetManagementPack(workItemMpName, Constants.mpKeyTocken, Constants.mpSMR2Version);
    ManagementPack mpSettings = emg.GetManagementPack(workItemSettingMpName, Constants.mpKeyTocken, Constants.mpSMR2Version);
    ManagementPack knowledgeLibraryMp = emg.GetManagementPack(ManagementPacks.knowledgeLibrary, Constants.mpKeyTocken, Constants.mpSMR2Version);

    ManagementPackClass workItemClass = emg.EntityTypes.GetClass(workItemClassName, workItemMp);
    ManagementPackClass workItemClassSetting = emg.EntityTypes.GetClass(workItemSettingClassName, mpSettings);

    EnterpriseManagementObject generalSetting = emg.EntityObjects.GetObject<EnterpriseManagementObject>(workItemClassSetting.Id, ObjectQueryOptions.Default);

    // here is the foreach loop that processes each class-object (in this case Incident) that was multi-selected in the view before executing the Task

    foreach (NavigationModelNodeBase node in nodes)
    {
        // we need to setup an IList which contains only 1 GUID that correspons to the Incident we are currently working on (node["Id"])
        // this is needed because we are using the IObjectProjectionReader.GetData(...) which gets an IList<Guid> as parameter in order to retreive the class-objects we want from the db

        IList<Guid> bmeIdsList = new List<Guid>();
        bmeIdsList.Add(new Guid(node[Constants.nodePropertyId].ToString()));

        // we are setting up the ObjectProjectionCriteria using the "System.WorkItem.Incident.ProjectionType" Type Projection as we need to get all the Relationships of the Incident
        // we will use ObjectRetrievalOptions.Buffered so that we don't get any data from the db which we don't need - we will only get the data one we call the IObjectProjectionReader.GetData(...) method
        // we are getting the data reader object using GetObjectProjectionReader(...) and setting its PageSize to 1 because we only need 1 object retrieved here

        ObjectProjectionCriteria incidentObjectProjection = new ObjectProjectionCriteria(incidentProjection);
        ObjectQueryOptions queryOptions = new ObjectQueryOptions(ObjectPropertyRetrievalBehavior.All);
        queryOptions.ObjectRetrievalMode = ObjectRetrievalOptions.Buffered;
        IObjectProjectionReader<EnterpriseManagementObject> incidentReader = emg.EntityObjects.GetObjectProjectionReader<EnterpriseManagementObject>(incidentObjectProjection, queryOptions);
        incidentReader.PageSize = 1;

        // we are using EnterpriseManagementObjectProjection for the Incident we are getting from the db instead of EnterpriseManagementObject
        // this is because we are getting a (Type) Projection object (class-object together with its Relationships & relationship class-objects

        EnterpriseManagementObjectProjection incident = incidentReader.GetData(bmeIdsList).FirstOrDefault();

        // we are doing the same for the new WorkItem class-object we are creating because we want to add Relationships (with their relationship class-objects from the Incident) here as well
        // if we would only have created the new WorkItem class and nothing else with it (no Relationships), we could have used the CreatableEnterpriseManagementObject class (which needs to be used when a new class-object is getting created)

        EnterpriseManagementObjectProjection workItem = new EnterpriseManagementObjectProjection(emg, workItemClass);

        // now we need to assign some Template to the new WorkItem (if a default/standard Template exists) in order to already have some Activities created
        // the Activities and all other Properties of the new WorkItem can be adjusted by modifying the default/standard Template for each WorkItem type

        if (!string.IsNullOrEmpty(workItemTemplateName))
        {
            ManagementPackObjectTemplateCriteria templateCriteria = new ManagementPackObjectTemplateCriteria(string.Format("Name = '{0}'", workItemTemplateName));
            ManagementPackObjectTemplate template = emg.Templates.GetObjectTemplates(templateCriteria).FirstOrDefault();

            if(template != null)
            {
                // if a Template with this name exists, we apply it to the new WorkItem by calling ApplyTemplate(...) on it

                workItem.ApplyTemplate(template);

                // if we have a Template, we also need to process each Activity that it contains in order to set the Prefix for each Activity (based on its type)
                // we are using the activityPrefixMapping variable we defined above in oder to map each Prefix based on each Activity class-type

                ManagementPack activityManagementMp = emg.GetManagementPack(ManagementPacks.activityManagementLibrary, Constants.mpKeyTocken, Constants.mpSMR2Version);
                ManagementPackRelationship workItemContainsActivityRelationshipClass = emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemContainsActivity, activityLibMp);
                ManagementPackClass activitySettingsClass = emg.EntityTypes.GetClass(ClassTypes.activitySettings, activityManagementMp);

                EnterpriseManagementObject activitySettings = emg.EntityObjects.GetObject<EnterpriseManagementObject>(activitySettingsClass.Id, ObjectQueryOptions.Default);

                // for each Activity that exists in the Template we applied, we are going to get its Prefix setting and apply it to its ID in the format: "PREFIX{0}"
                // "{0}" is the string pattern we need to set for any new WorkItem (including Activity) class-object we are creating as this will be replaced by the next ID available for the new WorkItem

                foreach (IComposableProjection activity in workItem[workItemContainsActivityRelationshipClass.Target])
                {
                    ManagementPackClass activityClass = activity.Object.GetClasses(BaseClassTraversalDepth.None).FirstOrDefault();
                    string prefix = activitySettings[null, activityPrefixMapping[activityClass.Name]].Value as string;
                    activity.Object[null, ActivityProperties.Id].Value = string.Format("{0}{1}", prefix, Constants.workItemPrefixPattern);
                }
            }
        }

        // we are setting the Properties for the new WorkItem class-object here from some Properties of the inital Incident (add more as needed)
        // it is of highest importance that we also set its status to New/Active (depending on WorkItem class-type) in order for it to be properly processed by the internal workflows on creation
        // if we don't set the the correct "creation" Status here, it will never be able to progress into a working state and will remain stuck in a "pending" state

        ManagementPackEnumerationCriteria workItemStatusNewEnumCriteria = new ManagementPackEnumerationCriteria(string.Format("Name = '{0}'", workItemStatusName));
        ManagementPackEnumeration workItemStatusNew = emg.EntityTypes.GetEnumerations(workItemStatusNewEnumCriteria).FirstOrDefault();

        workItem.Object[workItemClass, WorkItemProperties.Id].Value = string.Format("{0}{1}", generalSetting[workItemClassSetting, workItemSettingPrefixName], Constants.workItemPrefixPattern);
        workItem.Object[workItemClass, WorkItemProperties.Title].Value = string.Format("{0} ({1})", incident.Object[incidentClass, WorkItemProperties.Title].Value, incident.Object[incidentClass, WorkItemProperties.Id].Value);
        workItem.Object[workItemClass, WorkItemProperties.Description].Value = incident.Object[incidentClass, WorkItemProperties.Description].Value;
        workItem.Object[workItemClass, WorkItemProperties.Status].Value = workItemStatusNew.Id;

        // due to the fact that the Problem WorkItem does not have any Template we can use to create it, we need to handle this special case
        // we need to populate all the required fields when creating the Problem WorkItem, or creating it will fail (Urgency, Impact, Category)

        if (!string.IsNullOrEmpty(workItemUrgencyName))
        {
            ManagementPackEnumerationCriteria workItemUrgencyEnumCriteria = new ManagementPackEnumerationCriteria(string.Format("Name = '{0}'", workItemUrgencyName));
            ManagementPackEnumeration workItemUrgency = emg.EntityTypes.GetEnumerations(workItemUrgencyEnumCriteria).FirstOrDefault();
            workItem.Object[workItemClass, WorkItemProperties.Urgency].Value = workItemUrgency.Id;
        }

        if (!string.IsNullOrEmpty(workItemImpactName))
        {
            ManagementPackEnumerationCriteria workItemImpactEnumCriteria = new ManagementPackEnumerationCriteria(string.Format("Name = '{0}'", workItemImpactName));
            ManagementPackEnumeration workItemImpact = emg.EntityTypes.GetEnumerations(workItemImpactEnumCriteria).FirstOrDefault();
            workItem.Object[workItemClass, WorkItemProperties.Impact].Value = workItemImpact.Id;
        }

        if (!string.IsNullOrEmpty(workItemCategoryName))
        {
            ManagementPackEnumerationCriteria workItemCategoryEnumCriteria = new ManagementPackEnumerationCriteria(string.Format("Name = '{0}'", workItemCategoryName));
            ManagementPackEnumeration workItemCategory = emg.EntityTypes.GetEnumerations(workItemCategoryEnumCriteria).FirstOrDefault();
            workItem.Object[workItemClass, WorkItemProperties.Category].Value = workItemCategory.Id;
        }

        // we are adding the initial Incident to this new WorkItem as related WorkItem (System.WorkItemRelatesToWorkItem)

        ManagementPackRelationship workItemToWorkItemRelationshipClass = emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemRelatesToWorkItem, wiLibraryMp);
        workItem.Add(incident.Object, workItemToWorkItemRelationshipClass.Target);

        // we are closing the current Incident by setting its Status to Closed and setting a closed date

        incident.Object[incidentClass, IncidentProperties.Status].Value = incidentClosedStatus.Id;
        incident.Object[incidentClass, IncidentProperties.ClosedDate].Value = DateTime.Now.ToUniversalTime();

        // we create a new (analyst) comment (System.WorkItem.TroubleTicket.AnalystCommentLog) and we add it to the Incident in oder to comment the fact that it was closed tue to this WorkItem Transfomr Task

        CreatableEnterpriseManagementObject analystComment = new CreatableEnterpriseManagementObject(emg, analystCommentClass);
        analystComment[analystCommentClass, AnalystCommentProperties.Id].Value = Guid.NewGuid().ToString();
        analystComment[analystCommentClass, AnalystCommentProperties.Comment].Value = string.Format(Constants.incidentClosedComment, workItemClass.Name, workItem.Object.Id.ToString());
        analystComment[analystCommentClass, AnalystCommentProperties.EnteredBy].Value = EnterpriseManagementGroup.CurrentUserName;
        analystComment[analystCommentClass, AnalystCommentProperties.EnteredDate].Value = DateTime.Now.ToUniversalTime();

        ManagementPackRelationship incidentHasAnalystCommentRelationshipClass = emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemHasAnalystComment, wiLibraryMp);
        incident.Add(analystComment, incidentHasAnalystCommentRelationshipClass.Target);

        // we create an IList of RelationshipTypes we want to transfer from the Incident to the new WorkItem
        // this is the place we can add any new/custom RelationshipTypes which we want to transfer
        // just make sure that the RelationshipType can be transfered from an Incident to any other WorkItem class-type

        IList<ManagementPackRelationship> relationshipsToAddList = new List<ManagementPackRelationship>()
        {
            workItemToWorkItemRelationshipClass,
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.createdByUser, wiLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.affectedUser, wiLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.assignedToUser, wiLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemHasAttachment, wiLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemAboutConfigItem, wiLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemRelatesToConfigItem, wiLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.entityToArticle, knowledgeLibraryMp),
            emg.EntityTypes.GetRelationshipClass(RelationshipTypes.workItemHasCommentLog, wiLibraryMp),
        };

        // we are getting an instance of the "System.Membership" RelationshipType as we need to handle RelationshipTypes derived from it as a special case
        // the reason for this, is that Target class-objects of the "System.Membership" RelationshipType are bound to their Source class-objects
        // being bound, means that Target class-objects of membership relationships cannot belong to 2 different (Source) class-objects
        // because of this, we need to make a copy (using "CreatableEnterpriseManagementObject" to create a new object and copying the Properties) of the existing Target class-object
        // and add that to the new WorkItem instead of adding the already existing Target class-object

        ManagementPack systemLibraryMp = emg.GetManagementPack(ManagementPacks.systemLibrary, Constants.mpKeyTocken, Constants.mpSMR2Version);
        ManagementPackRelationship membershipRelationshipClass = emg.EntityTypes.GetRelationshipClass(RelationshipTypes.membership, systemLibraryMp);

        // we are going through each Target & Source Relationships of the Incident as defined in the relationshipsToAddList variable and adding them to the new WorkItem
        // we are handling the Target RelationshipTypes which are derived from "System.Membership" as a special case as explained above
        // notice that we are also removing these Relationships from the Incident by calling Remove()
        // we are removing the Relationships from the Incident for performance purposes - in order to have less Relationships (less data) in the db
        // comment the "itemProjection.Remove();" in order to keep the Relationships to the Incident as well if needed for some reason

        foreach (ManagementPackRelationship relationship in relationshipsToAddList)
        {
            if (incident[relationship.Target].Any())
            {
                foreach (IComposableProjection itemProjection in incident[relationship.Target])
                {
                    // create a new Target class-object (CreatableEnterpriseManagementObject) and add it to the projection as it is a member of a Membership RelationshipType (as explained above)
                    // notice that we DON'T remove such a Target class-object Relationship because it will also remove the class-object itself (because it is a Membership RelationshipType object and it cannot exist without this Relationship)
                    // we need it to exist because we are copying data from it and it needs to still exist in the db (ex. Attachments - we still need the binary data to exist in the db when we create the new Attachment object)
                    // we could of course delete it after we create the new WorkItem with its Relationships when calling "workItem.Overwrite()", but I chose not to do it

                    if (relationship.IsSubtypeOf(membershipRelationshipClass))
                    {
                        CreatableEnterpriseManagementObject instance = new CreatableEnterpriseManagementObject(emg, itemProjection.Object.GetClasses(BaseClassTraversalDepth.None).FirstOrDefault());
                        foreach (ManagementPackProperty property in itemProjection.Object.GetProperties())
                        {
                            instance[property.Id].Value = itemProjection.Object[property.Id].Value;
                        }

                        instance[null, Constants.entityId].Value = Guid.NewGuid().ToString();

                        workItem.Add(instance, relationship.Target);
                    }

                    // just add the existing Target object-class as it is not a member of a Membership RelationshipType (as explained above)

                    else
                    {
                        workItem.Add(itemProjection.Object, relationship.Target);
                        itemProjection.Remove();
                    }
                }
            }

            if(incident[relationship.Source].Any())
            {
                // we just create the new Relationship of the Source class-object to the new WorkItem because this is not affected by the Membership RelationshipType

                foreach (IComposableProjection itemProjection in incident[relationship.Source])
                {
                    workItem.Add(itemProjection.Object, relationship.Source);
                    itemProjection.Remove();
                }
            }
        }

        // this is where we actually save (write) the new data to the db, when calling "Overwrite()" - here saving the Incident we modified (set Status to Closed & deleted Relationships)
        // before we have just created the new objects and relationships in-memory
        // this is also the point when almost all of the code validation is being done
        // if there are any issues really creating/editing/adding these objects/realtionships, this is where we would get the errors

        incident.Overwrite();

        // we are want to handle the error here of saving the new WorkItem and its Relationships because we want to re-open the Incident in case there is an issue when creating the new WorkItem

        try
        {
            // this is where we actually save (write) the new data to the db, when calling "Overwrite()" - here saving the new WorkItem we created with its Relationships we added (from the Incident)

            workItem.Overwrite();
        }
        catch (Exception ex)
        {
            // if we faild to create the new WorkItem with its Relationships, we want to revert to setting the Incident to an Active Status (we re-open the Incident)

            ManagementPackEnumerationCriteria incidentActiveEnumCriteria = new ManagementPackEnumerationCriteria(string.Format("Name = '{0}'", EnumTypes.incidentStatusActive));
            ManagementPackEnumeration incidentActiveStatus = emg.EntityTypes.GetEnumerations(incidentActiveEnumCriteria).FirstOrDefault();

            incident.Object[incidentClass, IncidentProperties.Status].Value = incidentActiveStatus.Id;
            incident.Object[incidentClass, IncidentProperties.ClosedDate].Value = null;

            // again, after applying the new modifications in memory, we need to actually write them to the db using "Overwrite()"

            incident.Overwrite();

            // no need to show this because we are just passing it (throwing) to the wrapped try/catch block so it will be displayed and handled there

            throw ex;
        }
    }

    // if everything succeeds, we want to refresh the View (in this case, some View that shows Incidents as this is where we are calling our Task from)
    // we want to refresh the view to show the new Status of the Incient (as Closed in this case) - if the View only shows non-Closed Incidents, it will dissapear from the View

    RequestViewRefresh();
}
catch(Exception ex)
{
    // we want to handle all Exceptions here so that the Console does not crash
    // we also want to show a MessageBox window with the Exception details for troubleshooting purposes

    MessageBox.Show(string.Format("Error: {0}: {1}\n\n{2}", ex.GetType().ToString(), ex.Message, ex.StackTrace));
}

$mpName = "MPClassesHostingTest" # internal MP Name goes here
$message = "Perform the needed actions here (remove MP, re-import, etc.). After finishing the actions, press any key to continue."
$props = Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\System Center\2010\Common\Setup"
$instdir = $props.InstallDirectory
Add-Type -Path "$instdir`SDK Binaries\Microsoft.EnterpriseManagement.Core.dll"
$dataMap = @{}
$emg = New-Object Microsoft.EnterpriseManagement.EnterpriseManagementGroup("localhost")
$mp = Get-SCManagementPack -Name $mpName
# sort (descending) by Hosted classes first so that when we re-create the objects, we start re-creating the Hosted objects first
# this is needed because Hosted objects need to be created together with their HostING objects
# this way, when we create non-Hosted objects, we skip creating HostING objects which we already have created when creating their Hosted objects
$classList = Get-SCClass -ManagementPack $mp | ? { $_.Singleton -eq $false -and $_.Abstract -eq $false } | Sort-Object Hosted -Descending
$hostingRel = Get-SCRelationship -Name "System.Hosting"
foreach($class in $classList) {
    $instanceList = Get-SCClassInstance -Class $class
    foreach($instance in $instanceList) {
        $object = @{
            instance = $instance;
            rels = @{
                source = Get-SCRelationshipInstance -SourceInstance $instance | ? { $_.IsDeleted -eq $false -and $_.Name -ne "System.Hosting" };
                target = Get-SCRelationshipInstance -TargetInstance $instance | ? { $_.IsDeleted -eq $false -and $_.Name -ne "System.Hosting" };
            };
        }
        $dataMap.Add($instance.EnterpriseManagementObject.Id, $object)
    }
}
if($psISE) {
    Add-Type -AssemblyName System.Windows.Forms
    [System.Windows.Forms.MessageBox]::Show("$message")
} else {
    Write-Host "$message" -ForegroundColor Yellow
    $x = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
}
# Create all objects first so that we are sure we can use them for relationships afterwards
# These need to be all there created first in order (non-Hosted first & Hosted after)
foreach($key in $($dataMap.Keys)) {
    $entry = $dataMap.Item($key)
    $class = Get-SCClass -Name $entry.instance.EnterpriseManagementObject.GetClasses().Name
    $props = @{}
    if($class.Hosted -eq $true) {
        # need to handle Hosted classes differently (cannot use New-SCClassInstance cmdlet for various corner-cases)
        # handling with CreatableEnterpriseManagementObject instead (need to also create parent-Hosting class together with it)
        $hostingClass = $class.GetParentClasses([Microsoft.EnterpriseManagement.Configuration.DerivedClassTraversalDepth]::Recursive, $hostingRel, [Microsoft.EnterpriseManagement.Common.TraversalDepth]::OneLevel).Item(0)
        $hostingClassKey = $hostingClass.GetKeyProperties().Item(0)
        $object = New-Object -TypeName Microsoft.EnterpriseManagement.Common.CreatableEnterpriseManagementObject($emg, $hostingClass)
        $object.Item($hostingClass, $hostingClassKey.Name).Value = $entry.instance.$($hostingClassKey.Name)
        $object.Commit()
        $object = New-Object -TypeName Microsoft.EnterpriseManagement.Common.CreatableEnterpriseManagementObject($emg, $class)
        $object.Item($hostingClass, $hostingClassKey.Name).Value = $entry.instance.$($hostingClassKey.Name)
        foreach($property in $class.GetProperties([Microsoft.EnterpriseManagement.Configuration.BaseClassTraversalDepth]::Recursive)) {
            if($entry.instance.$($property.Name) -ne $null) {
                $object.Item($class, $property.Name).Value  = $entry.instance.$($property.Name)
            }
        }
        $object.Commit()
        $entry.instance = $object
    } elseif((Get-SCClassInstance -Id $entry.instance.EnterpriseManagementObject.Id) -eq $null) {
        foreach($property in $class.GetProperties([Microsoft.EnterpriseManagement.Configuration.BaseClassTraversalDepth]::Recursive)) {
            if($entry.instance.$($property.Name) -ne $null) {
                $props.Add($property.Name, $entry.instance.$($property.Name))
            }
        }
        $entry.instance = New-SCClassInstance -Class $class -Property $props -PassThru
    }
    $dataMap.Set_Item($key, $entry)
}
# Now that we know we have all objects created, we can safely create the Relationships
foreach($key in $dataMap.Keys) {
    $entry = $dataMap.Item($key)
    foreach($rel in $entry.rels.Item("source")) {
        $relType = Get-SCRelationship -Id $rel.RelationshipId
        $target = Get-SCClassInstance -Id $rel.TargetObject.Id
        if($target -ne $null) {
            New-SCRelationshipInstance -RelationshipClass $relType -Source $entry.instance -Target $target
        }
    }
    foreach($rel in $entry.rels.Item("target")) {
        $relType = Get-SCRelationship -Id $rel.RelationshipId
        $source = Get-SCClassInstance -Id $rel.SourceObject.Id
        if($source -ne $null) {
            New-SCRelationshipInstance -RelationshipClass $relType -Source $source -Target $entry.instance
        }
    }
}

• https://blogs.technet.microsoft.com/mihai/2013/08/13/tweaking-the-ad-and-cm-connectors-in-service-manager-2012/

select MPName, convert(xml, MPXML)
from ManagementPack
where MPName = 'Microsoft.EnterpriseManagement.ServiceManager.Connector.Sms2011'

select *
from LFX.DataTable
where DataName like '%CMv5_%'

• The ones without the “Cached_” prefix are the ones which are being used by the Data Provider (gets the data directly from the custom source – in this case, the SCCM site database)
• The ones that have “Cached_” prefix are the ones used by the Data Consumer that will get the already imported data (from the LFX Staging Tables) and import it into the instance space (only then will it be visible in the Console)

<Object Path="$Context/Path[Relationship='LFX!System.LinkingFramework.ConnectorEmbedsTables' SeedRole='Source' TypeConstraint='LFX!System.LinkingFramework.DataTable']$">
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/DataName$">Cached_CMv5_LogicalComputers</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/WatermarkField$">E.Lfx_Timestamp</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/WatermarkType$">0</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/BatchIdField$">E.Lfx_RowId</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/BatchIdType$">0</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/BatchIdSize$">500</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/UseCache$">false</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/QueryString$">
                                         SELECT E.Lfx_RowId,
                                                       E.Lfx_SourceID,
                                                       E.Lfx_Status,
                                                       E.DisplayName, 
                                                       E.PrincipalName, 
                                                       E.NetbiosComputerName, 
                                                       E.NetbiosDomainName,
                                                       E.OffsetInMinuteFromGreenwichTime,
                                                       E.IsVirtualMachine,
                                                       E.LastInventoryDate,
                                                       E.ActiveDirectorySite
                                         from [LFXSTG].v_Cached_CMv5_LogicalComputers E 
                                  </Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/CollectionName$">Cached_CMv5_LogicalComputers</Property>
       <Property Path="$Context/Property[Type='LFX!System.LinkingFramework.DataTable']/DependOnDataTable$">Cached_CMv5_PhysicalComputers</Property>
</Object>

CREATE VIEW [LFXSTG].[v_Cached_CMv5_LogicalComputers] AS
SELECT S.Lfx_RowId,
        S.Lfx_SourceID,
        S.Lfx_Timestamp,
        CCX.Lfx_Status,
        CCX.Name0 AS 'DisplayName', 
        COALESCE(CCX.Name0, S.Netbios_Name0) 
              + '.' + COALESCE(CCX.Domain0, S.Resource_Domain_OR_Workgr0) AS 'PrincipalName', 
        S.Netbios_Name0 AS 'NetbiosComputerName', 
        S.Resource_Domain_OR_Workgr0 AS 'NetbiosDomainName',
        CCX.CurrentTimeZone0 AS 'OffsetInMinuteFromGreenwichTime',
        CCX.IsVirtualMachine,
        W.LastHWScan AS 'LastInventoryDate',
        S.AD_Site_Name0 AS 'ActiveDirectorySite'
FROM LFXSTG.CMv5_SYSTEM S
CROSS APPLY 
LFXSTG.fn_CheckCMv5CachedComputers(S.Lfx_SourceID, S.ResourceID, S.SMS_Unique_Identifier0, S.Lfx_Status) AS CCX
    LEFT JOIN LFXSTG.CMv5_WORKSTATION_STATUS W
        ON S.ResourceID = W.ResourceID AND S.Lfx_SourceId = W.Lfx_SourceId
WHERE S.Netbios_Name0 IS NOT NULL 
    AND S.Resource_Domain_OR_Workgr0 IS NOT NULL
    AND CCX.Lfx_Status != 'I'

select *
from LFX.DataTable
where DataTableName = 'CMv5_WORKSTATION_STATUS'

SELECT
       S.ChangeAction as Lfx_Status,
       S.ResourceID,
       S.BatchingKey,
       S.GroupKey,
       S.TimeStamp,
       S.LastHWScan,
       S.SystemDefaultLCID,
       S.TimezoneOffset,
       S.LastReportVersion
FROM SCCM_Ext.vex_GS_WORKSTATION_STATUS S
INNER JOIN SCCM_Ext.vex_FullCollectionMembership CM
ON S.ResourceID = CM.ResourceID
INNER JOIN SCCM_Ext.vex_Collection C
ON C.CollectionID = CM.CollectionID
WHERE S.LastHWScan IS NOT NULL 
AND C.ChangeAction = 'U' and CM.ChangeAction = 'U'
AND $COLLECTIONLIST
ORDER BY S.rowversion

• Notice $COLLECTIONLIST here – this is replaced with the SCCM Site Collection filters that you have configured in your SCCM Connector.

• SQL Profiler trace on the SCCM Site database
• SQL Profiler trace on the ServiceManager database
• SM ETW Trace on LFX Connectors: cmd -l VER -a CON

select
       wss.Lfx_RowId,
       wss.Lfx_Timestamp,
       ds.DisplayName,
       wss.Lfx_Status,
       wss.ResourceID,
       wss.BatchingKey,
       wss.GroupKey,
       wss.TimeStamp,
       wss.LastHWScan,
       wss.SystemDefaultLCID,
       wss.LastReportVersion
from LFXSTG.CMv5_WORKSTATION_STATUS wss
join LFX.DataSource ds
       on wss.Lfx_SourceId = ds.DataSourceId
where wss.ResourceId in (RES_ID_1, RES_ID_2, RES_ID_3, RES_ID_ETC)

select
       cwt.ClientWorkTableId as [WorkTableId],
       cwt.Watermark as [WaterMark],
       cwt.LastSyncTime as [LastSyncTime],
       ds.DataSourceName as [ConnectorName],
       ds.DisplayName as [ConnectorDisplayName],
       dt.DataName as [DataName],
       dt.DataTableName as [DataTableName],
       (
              case
                     when dt.WatermarkType = -1 then 'None'
                     when dt.WatermarkType = 0 then 'DateTime'
                     when dt.WatermarkType = 1 then 'Timestamp'
                     when dt.WatermarkType = 2 then 'Number'
                     when dt.WatermarkType = 3 then 'Number'
              end 
       ) as [WaterMarkType]
from LFX.ClientWorkTable as cwt
inner join LFX.DataSource as ds
       on ds.DataSourceId = cwt.DataSourceId
inner join LFX.DataTable as dt
       on dt.DataTableId = cwt.DataTableId

• Repace “AD_CONNECTOR_DSIPLAYNAME” with the DisplayName of the (SCCM) Connector which you are interested in (as seen from the results of the previous query).
• Pay attention to the WaterMarkType and based on that, understand what type of value we need to set in order for the WaterMark to be set to “0” and try to re-sync everything related to this type of data (in this example CMv5_WORKSTATION_STATUS)

update cwt
set cwt.Watermark = 0x00000000
from LFX.ClientWorkTable as cwt
inner join LFX.DataSource as ds
       on ds.DataSourceId = cwt.DataSourceId
inner join LFX.DataTable as dt
       on dt.DataTableId = cwt.DataTableId
where
       ds.DisplayName = 'REPLACE_WITH_CONNECTOR_DSIPLAYNAME_STRING_HERE' and
       dt.DataName = 'CMv5_WORKSTATION_STATUS'

http://support.microsoft.com/kb/3039363

We are constantly improving SC Service Manager and we're not kidding about it ;)

The fixes and performance improvement in Update Rollup 6 are a proof of this!

Great job guys!

Enjoy!

http://blogs.msdn.com/b/microsoft_press/archive/2015/04/21/free-ebook-microsoft-system-center-operations-manager-field-experience.aspx

Check out the description - the contents are pretty interesting as we are explaining a lot of inside knowledge about deployment, optimizations, troubleshooting and internals of the product :D

Enjoy!

Beside other cool fixes, this UR includes also some cool changes for the Data Warehouse component. These will make it much easier to troubleshoot possible issues that may appear.

This is part of our effort to improve the Data Warehouse and the Service Manager product in general. I hope you will like it and await more good news in the future :D

INFOS:

Note The System Center 2012 R2 Service Manager Update Rollups are cumulative. Therefore, this update rollup contains new fixes for the following issues along with fixes that were included in System Center 2012 R2 Service Manager Update Rollup 4, Update Rollup 3, and Update Rollup 2.

• Change request still stays "in progress" when last activity is skipped and all previous activities are completed.
  
  
• ActualStartDate and ActualEndDate field values for Change Request and Release Request do not set.
  
  
• Multiple System Center Service Manager Connectors to System Center Configuration Manager execute and complete successfully. However, the start, finish, and status property values are not updated in the Service Manager Console under the Connectors view, and event 3334 is logged.
  
  
• Trying to delete a Service Request template that is being used by a Request Offering causes a cryptic and non-user-friendly error message.
  
  
• The "Group By" functionality in a view does not work correctly when the Group By column has empty fields.
  
  
• The console crashes when the user tries to open an already opened attachment from a Work Item form.
  
  
• Opening the Views for Groups takes a long if there are a large number of explicit members in the groups.
  
  
• The OM CI connector fails if you try to import a distributed application that contains an instance of the Hyper-V Virtual Network Adapter class.
  
  
• Monitoring Host process crashes if there exists a notification after the user who created it was deleted or moved to a hidden organizational unit (OU) in Active Directory.
  
  
• An exception is raised if a user tries to open a sorted view that is created by using TypeProjection on a custom class that is neither Abstract and nor a first Concrete class.
  
  
• An exception is thrown when a user tries to open the Service Components tab on the Service Maps form when there are large number of services in CMDB.
  
  
• Enhanced in-event logging for DW jobs.
  
  Logging the batch start and completion events for all DW job categories in event log. The start and completion event will include the following:
  • Process category
  • Process name
  • Batch ID
  • Batch start or completion time
  
  
  
• Enhanced in event logging for DW cube processing.
  • Time taken by Cube’s batch ID to complete will be logged in seconds.
  • Information about whether the cube is processed under Analysis Services that are running in Microsoft SQL Server Standard Edition or in SQL Server Enterprise Edition.
  • Event logs will be added during processing of each Dimension and Measure Group for each partition (if applicable) for both enterprise and standard editions of SQL Server. 
  
  
  
• Added the new Windows PowerShell cmdlet Get-SCDWInfraLocations on the Service Manager management server to retrieve the following location information about its data warehouse infrastructure:
  • Service Manager database
  • Service Manager data warehouse database
  • Service Manager Analysis server database
  • Data Warehouse Reporting Server database and Report Server URL

It is important to first of all understand how the DW Jobs work. There are some Jobs that are only visible by looking into the database, others may be seen in PowerShell only and the most basic ones can be seen in the Console. From the perspective of the databases, the DW Jobs are actually called Processes. From now on, I will refer to the DW Jobs as Processes in this article.

The DW Processes are actually categorized like this:

• Process Categories - this is top level way of categorizing Processes (Jobs)
• Processes - these are the actual Processes that exist (ex. DWMaintenance, MPSyncJob, Extract, Transform, Load, etc.)
• ProcessModules - each Process has several ProcessModules which it will actually execute in order to "finish" - there is also a certain order in which these will run depending on dependencies on one another and the level of the modules

Now what you need to know is that each Process will be executed via a Schedule which is actually a Rule (Workflow that runs on a schedule and starts its owning Process). When a Process is scheduled (for execution) it will actually get a new Batch created. Think of a Process as a class and of a Batch as an instance of that class that is actually doing something. It is the same for a ProcessModule, each will get a WorkItem when it is scheduled for execution. Because a Process can have from one to many ProcessModules, when a Batch is created, it also gets all the WorkItems associated with it (ProcessModules of the Process).

So, to recap, because we will be using these a lot in this article:

• Batch = an instance of a Process (which in the Console or PowerShell is a Job)
• WorkItem = an instance of a ProcessModule of a Process

Important to know and also the way the processing/execution will depend on and work, is that each Batch, as well as each WorkItem (of a Batch) will have a certain Status :

• Success (1) - the Batch/WorkItem was successful and at that point the actual Status gets changed to Complete
• Failed (2) - there was an error in the Batch/WorkItem and it will be retried on the next execution
• Not Started (3) - the Batch/WorkItem was scheduled for execution and it will be executed on the next interval (schedule) - this is a good Status, it means there was no error and it will run the next time it needs to
• Running (4) - the Batch/WorkItem is currently running
• Stopped (5) - the Batch/WorkItem was stopped (by the user from the Console - Suspend or PowerShell Stop-SCDWJob)
• Completed (6) - the Batch/WorkItem has successfully completed without any errors
• Waiting (7) - the Batch/WorkItem is waiting for other Batches/WorkItems to finish first on which they are dependent

It is important to know that the actual Status is important for the WorkItems, because the Batch Status depends directly on the Statuses of its WorkItems - here are some examples:

• A Batch can have the Status set to Running while all its WorkItems still have the Status of Not Started => this is not a bad thing, it just means that the certain Batch has received a "start request" from the schedule, but the (first) WorkItem(s) cannot run yet because they are waiting on another Batch/WorkItem (of a different Process) to finish first
• If one of the WorkItems of a Batch will fail and have a Status of Failed, then the Batch itself will get the Status of Failed => all WorkItems need to complete successfully and thus have a Status of Completed for the Batch to also get the Status of Completed

When a Process gets created the first time (on installation or DW registration), a new Batch will be created with some BatchId (the newest one possible - starting from 1 - for the first Process). Then for each ProcessModule of this Process, a new WorkItem with a certain WorkItemId will get created for this corresponding Batch. This will all initially have the Status of Not Started and when the corresponding Schedule Rule will reach it schedule it will send a "start request" for it's Batch. This will set the Status of that Batch to Running. The actual way how these run is that there is a Rule associated with each Process and these Rules run every 30 seconds. As soon as such a Rule runs, if it sees that its corresponding Process has the (latest) Batch with a Status of Running, will take the first/next WorkItem that has a Status of Not Started (depending on dependencies and level) and will try to execute that WorkItem. If it can execute and it is successful, it will get the Status of Completed once it finishes. If it fails, then it will get the Status of Failed. It is important to know here, that it can and will get a Status of Failed also without there being an actual error - but rather it cannot start/run right now because a dependent WorkItem is not finished yet and thus the "error" is actually "waiting on workitems to finish".

Another very important thing to know is that there is also a certain synchronization method between the different Processes, so that they don't interfere with each others work. This is needed because there are some actions which change the database(s) schema, add/remove columns from tables, indexes, primary/foreign keys, etc. We would not want the ETL (Extract, Transform, Load) Processes to run and copy/transform data while we are doing such changes. Because of this, we have these implementations:

• When the DWMaintenance and/or MPSyncJob Processes start running, the first thing they will do, is disable the ETL (Extract, Transform, Load) Processes => it is very important to note here that if either of the ETL Processes will be running when they will get disabled, these will *still* show the Status of Running, but in reality they will not run because they get disabled.
• Because DWMaintenance and MPSyncJob Processes should also not be running at the same time, these use a locking mechanism - whichever of them gets this lock first will be allowed to run until it finishes (Status of Completed) while the other one will remain waiting for it to finish, even if the Status will still show as Running and even if it will *not* be directly disabled => there is a table in the DWStagingAndConfig database used for holding the lock which you can look at - it is called LockDetails.

I have talked about some "invisible" Jobs which can only be seen by querying the database. There are actually a lot of those, but the "real" way to talk about them, is to go back to the Process Categories. There is a very important Process Category here which needs to be mentioned, which is the Deployment category. This is invisible because the MPSyncJob actually takes care handing work over this this category. MPSyncJob will associate Management Packs from one data source to another (ex. from CMDB to the DW). Each Management Pack that got associated will get a new Job (Process) created for it which falls under the Deployment Process Category. We can also refer to this as the "Deployment Job" if you will - and if fact, this is how you would usually hear about this. These are responsible for *actually* deploying (or installing if you will) the Management Packs which were synchronized over - without this, nothing would work because we would have no extended information in the Data Warehouse about what Classes, Relationships, etc. we have in the CMDB (and other managed data sources which we can register to the Data Warehouse).

If either of the deployment "jobs" fail or are not yet finished, then you can be more or less sure that nothing will work properly and you will 99% get errors related to either of the ETL (Extract, Transform, Load) Processes and maybe not only those - depending on where deployment is at that point.

While any of the Deployment Processes will follow the same rules as the others, that is get a Batch and one to many WorkItems, it also has another way of actually making the deployment. Each such Deployment Process (for each Management Pack) will have a new DeploySequence created (with a DeploySequenceId). Because each Management Pack has one to many "items" (Classes, Relationships, etc.) which need to be deployed, for each DeploySequence we will have one to many DeployItems created. This is important information for when you get into the actual troubleshooting part in the database.

In case you are not familiar with the basics of troubleshooting the DW Jobs from PowerShell, I suggest first starting with this article: http://technet.microsoft.com/en-us/library/hh542403.aspx

Also, if you are not familiar with what the ETL (Extract, Transform, Load) Jobs are and how these work, I suggest reading this article as well: http://blogs.technet.com/b/servicemanager/archive/2009/06/04/data-warehouse-anatomy-of-extract-transform-load-etl.aspx

Remember to always start by checking if there are any Management Packs where they Deployment Status shows as Failed in the SvcMgr Console in the Date Warehouse tab under the Management Packs view. For any such Management Pack which is Failed, you should run the "Restart Deployment" task from the Console in the tasks pane. If you are lucky, it was just some timeout or deadlock and it will succeed this time. If not, then you can always see errors about this failure as well as any other failures in the DW Jobs by looking into the Operations Manager event log directly on the SvcMgr Data Warehouse Management Server (filter by sources: Data Warehouse and Deployment).

Additionally, you can try to force reset the DW Jobs and run them in a certain order by using the script from this article: http://blogs.technet.com/b/mihai/archive/2013/07/03/resetting-and-running-the-service-manager-data-warehouse-jobs-separately.aspx

The Data Warehouse database which we are interested in when troubleshooting the DW Jobs is the DWStagingAndConfig database. Here is a list of tables of interest when troubleshoot the DW Jobs and at the end, also a file attached with useful sql queries:

1. The Infra.ProcessCategory table is where all the existing Process Categories are stored. It has a column called IsEnabled which needs to be 1 in order for any Process under this Process Category to be able to run - this is only modified (0 and 1) by the DWMaintenance and/or MPSyncJobs when they run - this is how they disable the other Processes.

2. The Infra.Process table is where all existing Processes are available and classified on Process Categories via the ProcessCategoryId column. These also have a IsEnabled column which should always be 1 in order for them to be able to execute. The only reason why either of these would have IsEnabled set to 0 is if anyone explicitly disabled these by using the Disable-SCDWJob CMDLet, which should never be done. To disable a Process (Job) always disable only its Schedule by using Disable-SCDWJobSchedule. In the screenshot below, I have explicitly filtered for only the "common" Processes, but if you will query the entire table, you will see all of them.

3. The Infra.ProcessModule table is where all existing ProcessModules are located and are classified on each Process via the ProcessId column. That is what you should use in a where clause of a sql query to see all ProcessModules that belong to a certain Process. Here is an example in the screenshot below for all ProcessModules of the DWMaintenance Process. These are not "ordered" in this result - if you want to figure out the "order" in which they would run, you need to check out the ModuleLevel column along with each dependency for each module which can be seen in the Infra.ModuleTriggerCondition table.

4. The Infra.Batch table is where all the Batches will be found (current/previous and next - check out Infra.BatchHistory for a history of these). This is where you will be able to see the BatchIds for the Processes and you can view them for a specific Process by using the ProcessId in a sql where clause. In the example screenshot below, we can see the Batches of the time the screenshot was taken for the Process with ProcessId = 1 (which in this case is the DWMaintenance Process).

5. The Infra.WorkItem table is where all the WorkItems of a Batch will be found. The most important part here is that this table is where we can see any errors for failed WorkItems of a Batch (that is an instance of a certain Process). In the screenshot below, an example of the WorkItems of the Batch with BatchId 3098 that is an instance of the DWMaintenance Process.

6. The DeploySequence table where we have an entry for each DeploySequence (so each MP that is or was deployed). You should know that in this process, a staging table is used which is called DeploySequenceStaging and if deployment finished and was successful, then this staging table should be *empty*. Here is a screenshot (not all results) of how this looks like.

7. The DeployItems table where we store an entry for each DeploySequence (so each MP that is or was deployed). You should know that in this process, a staging table is used which is called DeployItemStaging and if deployment finished and was successful, then this staging table should be *empty*. Here is a screenshot with the list of DeployItems belonging to the DeploySequence of the System.WorkItem.Incident.Library Management Pack.

In addition to that, you can always use SQL Server Profiler to get detailed information about what queries are being executed, details into different errors that might happen and of course having the queries, also a possibility of understanding the "why" behind them. The most useful column on which you can filter on is the ApplicationName column a the trace. These are the various application names used by the DW Jobs processing modules:

• DW Jobs scheduling, execution and processing: SC DAL--Orchestration and SC DAL--SCDW
• DWMaintenance job: SC DAL--Maintenance
• MPSync job: SC DAL--MP Sync
• Extract jobs: SC DAL--SCDW Extract Module
• Transform job: SC DAL--SCDW Transform Module
• Load jobs: SC DAL--SCDW Load Module
• Cube related jobs: Microsoft SQL Server Analysis Services (and general troubleshooting of SQL Analysis Services – Application event log and SQL Analysis Services tracing)

A cool template I usually use for the SQL Profiler Trace which is in 99% of the cases enough, has these settings (events and columns):

NOTE: Attached, you will also find a file (DWJobs_sql_queries.sql) that contains (commented) useful sql queries which you can and should use when troubleshooting the DW Jobs.

Good hunting! :)

DWJobs_sql_queries.sql

http://support.microsoft.com/kb/2992012

I encourage you to install UR4 for Service Manager because next to other fixes, there's a very important fix included here for the Data Warehouse.

This is part of the fixes and improvements we are trying to bring to the Data Warehouse: http://support.microsoft.com/kb/2989601

Data Warehouse Transform jobs have a hardcoded 60-minute time-out. Therefore, Data Warehouse jobs cannot be disabled for very long because the volume of data to be transformed can quickly pile up. This can cause an issue if the volume exceeds the amount that can be processed by the transform modules within the time-out period.

This fix included in UR4 has 2 parts: an improvement of an involved stored procedure which makes it be able to run faster + a new registry setting that can be created to be able to control the default timeout for the command, as well as changing the default from 60 to 180 seconds.

Happy updating! :D