long startup time of AX32 .exe
Last week I was on call to assist a customer with t-shooting a long startup time of AX32.exe performance issue – in my client’s environment Ax32.exe took around 18 seconds to fully show up the UI – this happened to some users only- it sounds like either GPO or those users’ profiles have a role in this.
After capturing some netmon packets from the AX client machine, I notice an interesting access pattern: HTTP communications are first attempted to an external IP before the real RPC packets are sent out to AOS port . As we know normally RPC shall be the default transportation channel, unless SharePoint integration, or custom codes referencing HTTPs. However, after confirming with my customer’s developers none has been the case – neither custom codes nor Sharepoint has been configured to bind to this HTTP web site.
I decide to capture some memory snapshots to better understand how the 18 seconds are used. Here is the instruction:
C:\dump>procdump -s 5 -n 3 –ma ax32.exe ax32.dmp
This command is going to write 3 memory snapshots every 5 seconds apart. The ProcDump is from https://technet.microsoft.com/en-us/sysinternals/dd996900
The timing to run the above command is very important: immediately after double click AX32.exe.
With the memory snapshots, there are some interesting observations under a debugger like WinDBG:
1. AX32.exe is attempting to communicate with https://crl.microsoft.com/pki/crl/products/CSPCA.crl with a timeout of 15 seconds.
2. The HTTP communication is triggered by an attempt to load a .NET DLL (e,g: c:\program files\Microsoft Dynamics AX\50\Client\Bin\Microsoft.Dynamics.Kernel.Client.dll, c:\program files\Microsoft Dynamics AX\50\Client\Bin\Microsoft.Dynamics.Xml.dll).
0:000> kcnL
#
00 ntdll!KiFastSystemCallRet
01 ntdll!ZwWaitForSingleObject
02 kernel32!WaitForSingleObjectEx
03 cryptnet!CryptRetrieveObjectByUrlWithTimeout
04 cryptnet!CryptRetrieveObjectByUrlW
05 cryptnet!RetrieveObjectByUrlValidForSubject
06 cryptnet!RetrieveTimeValidObjectByUrl
07 cryptnet!CTVOAgent::GetTimeValidObjectByUrl
08 cryptnet!CTVOAgent::GetTimeValidObject
09 cryptnet!FreshestCrlFromCrlGetTimeValidObject
0a cryptnet!CryptGetTimeValidObject
0b cryptnet!GetTimeValidCrl
0c cryptnet!GetBaseCrl
0d cryptnet!MicrosoftCertDllVerifyRevocation
0e crypt32!VerifyDefaultRevocation
0f crypt32!CertVerifyRevocation
10 crypt32!CChainPathObject::CalculateRevocationStatus
11 crypt32!CChainPathObject::CalculateAdditionalStatus
12 crypt32!CCertChainEngine::CreateChainContextFromPathGraph
13 crypt32!CCertChainEngine::GetChainContext
14 crypt32!CertGetCertificateChain
15 wintrust!_WalkChain
16 wintrust!WintrustCertificateTrust
17 wintrust!_VerifyTrust
18 wintrust!WinVerifyTrust
19 mscorsec!GetPublisher
1a mscorwks!PEFile::CheckSecurity
1b mscorwks!PEAssembly::DoLoadSignatureChecks
1c mscorwks!PEAssembly::PEAssembly
1d mscorwks!PEAssembly::DoOpenHMODULE
1e mscorwks!PEAssembly::OpenHMODULE
1f mscorwks!AppDomain::BindExplicitAssembly
20 mscorwks!AppDomain::LoadExplicitAssembly
21 mscorwks!ExecuteDLLForAttach
22 mscorwks!ExecuteDLL
23 mscorwks!CorDllMainForThunk
24 mscoree!CorDllMainWorkerForThunk
25 mscoree!ShellShim_CorDllMainWorkerForThunk
26 mscoree!VTableBootstrapThunkInitHelper
27 mscoree!VTableBootstrapThunkInitHelperStub
28 Ax32!InitClientBridge
29 Ax32!OpenClientBridge
2a Ax32!GetClientBridge
2b Ax32!xApp::Create
2c Ax32!AxWnd::AxApp::CreateAppWindow
2d Ax32!AxWnd::AxApp::main
2e Ax32!wWinMain
2f Ax32!__tmainCRTStartup
30 kernel32!BaseThreadInitThunk
31 ntdll!__RtlUserThreadStart
32 ntdll!_RtlUserThreadStart
0:000> .frame 3
03 0022ea98 71e71f7c cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x1a5
0:000> dv
pwszUrl = 0x040e90a4 "https://crl.microsoft.com/pki/crl/products/CSPCA.crl"
pszObjectOid = 0x00000002 "--- memory read error at address 0x00000002 ---"
dwRetrievalFlags = 0x202005
dwTimeout = 0x3a98
ppvObject = 0x0022ec44
pAuxInfo = 0x0022ebe8
hCertDiagEvent = 0x00000000
dwThreadId = 0x3898
hThread = 0x00000000
dwErr = 0x71e880dc
fResult = 0n104
hToken = 0x00000000
0:000> ?? 0x3a98
int 0n15000
0:000> .frame 19
19 0022f5b8 70accabd mscorsec!GetPublisher+0xe4
0:000> dv
pwsFileName = 0x01a31bf0
hFile = 0x00000000
dwFlags = 3
pInfo = 0x0022f5f0
dwInfo = 0x0022f5f8
sWTD = struct _WINTRUST_DATA
gV2 = struct _GUID {31d1adc1-d329-11d1-8ed8-0080c76516c6}
sCorPolicy = struct _COR_POLICY_PROVIDER
sWTFI = struct WINTRUST_FILE_INFO_
0:000> dc 0x01a31bf0
01a31bf0 003a0043 0050005c 006f0072 00720067 C.:.\.P.r.o.g.r.
01a31c00 006d0061 00460020 006c0069 00730065 a.m. .F.i.l.e.s.
01a31c10 004d005c 00630069 006f0072 006f0073 \.M.i.c.r.o.s.o.
01a31c20 00740066 00440020 006e0079 006d0061 f.t. .D.y.n.a.m.
01a31c30 00630069 00200073 00580041 0035005c i.c.s. .A.X.\.5.
01a31c40 005c0030 006c0043 00650069 0074006e 0.\.C.l.i.e.n.t.
01a31c50 0042005c 006e0069 004d005c 00630069 \.B.i.n.\.M.i.c.
01a31c60 006f0072 006f0073 00740066 0044002e r.o.s.o.f.t...D.
0:000> dc
01a31c70 006e0079 006d0061 00630069 002e0073 y.n.a.m.i.c.s...
01a31c80 0065004b 006e0072 006c0065 0043002e K.e.r.n.e.l...C.
01a31c90 0069006c 006e0065 002e0074 006c0064 l.i.e.n.t...d.l.
01a31ca0 0000006c 00650052 5772b2b5 8800006c l...R.e...rWl...
01a31cb0 00000002 00000001 01a31d20 01a83461 ........ ...a4..
01a31cc0 00000002 01a8b8f5 00000071 01a8b8f7 ........q.......
01a31cd0 9c534700 01bfca24 dfeb0700 01d6366f .GS.$.......o6..
01a31ce0 00000071 01a8b988 01a83429 00000002 q.......)4......
Our next question is: how to explain the above behaviors we are seeing?
The URL being triggered https://crl.microsoft.com/pki/crl/products/CSPCA.crl is actually used by CRL (Certificate Revocation List). The .NET implementation of CRL verification is based on an assembly’s Authenticode signature (please differentiate this with .NET strong name signature). AX32.exe refers to some .NET-based assemblies like c:\program files\Microsoft Dynamics AX\50\Client\Bin\Microsoft.Dynamics.Kernel.Client.dll, c:\program files\Microsoft Dynamics AX\50\Client\Bin\Microsoft.Dynamics.Xml.dll, and these assemblies are both strong-name signed and Authenticode signed.
For detailed explanation into the RCL and Authenticode please refer to:
So here is the fix:
As in my client’s environment accessing to internet is limited, and their DNS is not configured to further resolve the name to crl.microsoft.com, so we recommend to completely turn off the CRL for ax32.exe.
Basically KB947988 suggests either turning off CRL on a machine basis or enabling internet connection to https://crl.microsoft.com.
1. Turning off CRL is a good option for those in a reliable/hardened environment by turning off the “Software Restriction Policies” (https://technet.microsoft.com/en-us/library/bb457006.aspx#EAAA), or by by unchecking an option from Internet Explorer, “Tools | Options |Advanced | Security |Check for publisher’s certificate revocation”.
2. Enabling internet connection to https://crl.microsoft.com means to ensure the DNS naming resolution fine on crl.microsoft.com.
3. A better fix with restricted impact on other applications: Apart from the above 2 options we can also disable the CRL checking for AX32.exe alone – this is going to only impact AX32.exe as we are pretty sure all .NET assemblies being loaded by AX32.exe are reliable and fully trusted. This can be done:
a) Create a new text-base file named c:\program files\Microsoft Dynamics AX\50\Client\Bin\ax32.exe.config
b) The content for ax32.exe.config shall look like below:
<configuration>
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
</configuration>
thanks
Clifford