iSCSI Storage ETW tracing
Applies to:
Windows Server 2012 R2
Hyper-V 2012 R2
If you are troubleshooting an iSCSI storage problem, then these are the data that you should consider collecting.
The first item to consider is:
Using file copy to measure storage performance – Why it’s not a good idea and what you should do instead
https://blogs.technet.microsoft.com/josebda/2014/08/18/using-file-copy-to-measure-storage-performance-why-its-not-a-good-idea-and-what-you-should-do-instead/
#2) Setup a perfmon
Setting a local perfmon in a Windows client or Windows Server.
https://blogs.technet.microsoft.com/yongrhee/2015/05/16/setting-a-local-perfmon-in-a-windows-client-or-windows-server/
or
Setting a local perfmon in 64-bit Hyper-V 2012 R2 and/or 64-bit Windows Server 2012 R2 Hyper-V.
https://blogs.technet.microsoft.com/yongrhee/2016/11/19/setting-a-local-perfmon-in-64-bit-hyper-v-2012-r2-andor-64-bit-windows-server-2012-r2-hyper-v/
#3) Setup a network trace
Network tracing (packet sniffing) built-in to Windows and Windows Server.
https://blogs.technet.microsoft.com/yongrhee/2012/12/01/network-tracing-packet-sniffing-built-in-to-windows-server-2008-r2-and-windows-server-2012/
or
:: Start Network tracing (a.k.a. sniffing) tracing on Hyper-V 2012 R2 or Windows Server 2012 R2 Hyper-V
netsh.exe trace start scenario=AddressAcquisition,LAN,NDIS,NetConnection,Virtualization capture=yes capturetype=both report=yes persistent=no maxSize=4096 fileMode=circular correlation=no perfMerge=yes overwrite=yes traceFile=c:\Temp\NetTrace.etl
#4) Installing the tools for ETW tracing:
Download the latest Windows 10 SDK
Download: Windows 10 Software Development Kit (SDK)
https://blogs.technet.microsoft.com/yongrhee/2015/08/11/download-windows-10-software-development-kit-sdk/
and then install just the “Windows Performance Toolkit”
WPT: Installing the Windows Performance Toolkit v5.0 (WPRUI, WPR, Xperf)
https://blogs.technet.microsoft.com/yongrhee/2012/11/23/wpt-installing-the-windows-performance-toolkit-v5-0-wprui-wpr-xperf/
Note: Just be aware that the screen shots showcase the Windows 8.1 SDK, but you want the Windows 10 SDK.
#5) Setting up and starting the ETW tracing:
:: Start Kernel tracing and MiniFilter tracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -on PROC_THREAD+LOADER+FileIO+DISPATCHER+CSWITCH+DISK_IO_INIT+PROFILE+FILENAME+DRIVERS+NETWORKTRACE+DPC+INTERRUPT+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE -stackwalk CSwitch+ReadyThread+DiskReadInit+DiskWriteInit+DiskFlushInit+FileCreate+FileCleanup+FileClose+FileRead+FileWrite+FileFlush+MiniFilterPreOpInit+MiniFilterPostOpInit -f c:\temp\Pre_Kernel.etl -BufferSize 1024 -MinBuffers 1024 -MaxBuffers 1024 -MaxFile 4096 -FileMode Circular
::Start iSCSI tracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start iSCSITrace -on msiscsi_iScsi:0xffffffffffffffff:0xffffffff:'stack'+1babefb4-59cb-49e5-9698-fd38ac830a91:0xffffffffffffffff:0xffffffff:'stack' -f c:\temp\Pre_iSCSITrace.etl -BufferSize 512 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -FileMode Circular
::Start MPIOtracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start MPIOTrace -on 8E9AC05F-13FD-4507-85CD-B47ADC105FF6:0xffffffffffffffff:0xffffffff:'stack' -f c:\temp\Pre_MPIOTrace.etl -BufferSize 512 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -FileMode Circular
::Start DSM tracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start DSMTrace -on DEDADFF5-F99F-4600-B8C9-2D4D9B806B5B:0xffffffffffffffff:0xffffffff:'stack' -f c:\temp\Pre_DSMTrace.etl -BufferSize 512 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -FileMode Circular
::Start Storport tracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start Storport -on Microsoft-Windows-StorPort:0xffffffffffffffff:0xffffffff:'stack' -f c:\temp\Pre_Storport.etl -BufferSize 512 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -FileMode Circular
::Start StoragePerformance tracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start StoragePerformance -on A103CABD-8242-4A93-8DF5-1CDF3B3F26A6:0xffffffffffffffff:0xffffffff:'stack'+F5D05B38-80A6-4653-825D-C414E4AB3C68:0xffffffffffffffff:0xffffffff:'stack'+6B4DB0BC-9A3D-467D-81B9-A84C6F2F3D40:0xffffffffffffffff:0xffffffff:'stack'+C4636A1E-7986-4646-BF10-7BC3B4A76E8E:0xffffffffffffffff:0xffffffff:'stack'+CB587AD1-CC35-4EF1-AD93-36CC82A2D319:0xffffffffffffffff:0xffffffff:'stack'+cc7b00d3-75c9-42cc-ae56-bf6d66a9d15d:0xffffffffffffffff:0xffffffff:'stack'+90742496-54a9-4075-8079-425399080068:0xffffffffffffffff:0xffffffff:'stack' -f c:\temp\Pre_Storage_Performance.etl -BufferSize 512 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -FileMode Circular
::Start Microsoft-Windows-TCPIP tracing
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start TCPIP -on Microsoft-Windows-TCPIP:0xffffffffffffffff:0xffffffff:'stack' -f c:\temp\Pre_TCPIP.etl -BufferSize 512 -MinBuffers 320 -MaxBuffers 320 -MaxFile 4096 -FileMode Circular
::Start Hyper-V
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -start Hyper-V -on BB510E5F-2EB9-491A-81E4-F04654388F2B:0xffffffffffffffff:0xffffffff:'stack'+67E605EE-A4D8-4C46-AE50-893F31E13963:0xffffffffffffffff:0xffffffff:'stack'+67DC0D66-3695-47C0-9642-33F76F7BD7AD:0xffffffffffffffff:0xffffffff:'stack'+F2E2CE31-0E8A-4E46-A03B-2E0FE97E93C2:0xffffffffffffffff:0xffffffff:'stack'+29F239D9-979B-4160-A7DD-C4EA36A75181:0xffffffffffffffff:0xffffffff:'stack'+EEBBE72A-286C-4CCD-BBAD-88957BF388C4:0xffffffffffffffff:0xffffffff:'stack'+90742496-54A9-4075-8079-425399080068:0xffffffffffffffff:0xffffffff:'stack'+177D1599-9764-4E3A-BF9A-C86887AADDCE:0xffffffffffffffff:0xffffffff:'stack'+7B0EA079-E3BC-424A-B2F0-E3D8478D204B:0xffffffffffffffff:0xffffffff:'stack' -BufferSize 1024 -MinBuffers 1024 -MaxBuffers 1024 -f c:\temp\Pre_Hyper-V.etl
::The Hyper-V is optional, add it if you have the Hyper-V role installed.
#6) Reproduce the issue
::<repro>
Timeout.exe /t 300
Note: 300 is in seconds. 300 seconds is 5 minutes.
Note 2: Press Enter to stop
#7) Stop the ETW traces:
::Stop all the iSCSI and Storage related traces
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -stop -stop iSCSITrace -stop DSMTrace -stop MPIOTrace -stop Storport -stop StoragePerformance -stop TCPIP -stop Hyper-V -d c:\temp\Pre_iSCSI_Disk_Troubleshooting.etl::Stop Network tracing tracing
netsh.exe trace stop::Merge the WPP (Classic) based ETW traces
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Xperf.exe" -merge c:\temp\Pre_iSCSITrace.etl c:\temp\Pre_MPIOTrace.etl c:\temp\Pre_DSMTrace.etl c:\temp\Pre_Hyper-V.etl c:\temp\Pre_iSCSI_Disk_Troubleshooting.etl c:\temp\iSCSI_Classic-WPP_ETW.etl:: Stop the perfmon logs
logman.exe stop %ComputerName%_long_interval
logman.exe stop %ComputerName%_short_interval
#8) Analyzing the data:
Measuring Disk Latency with Windows Performance Monitor (Perfmon)
https://blogs.technet.microsoft.com/askcore/2012/02/07/measuring-disk-latency-with-windows-performance-monitor-perfmon/Hyper-V Performance – Storage
https://blogs.technet.microsoft.com/neales/2016/11/07/hyper-v-performance-storage/Disk Performance Internals
https://blogs.technet.microsoft.com/supportingwindows/2014/12/09/disk-performance-internals/Deciphering Storport Traces 101
https://blogs.technet.microsoft.com/askcore/2014/08/19/deciphering-storport-traces-101/