Well I haven't really posted in a while but I have good reasons, sort of. March went by quickly as I had to finish up an MBA class and moved apartments across the city of Chicago. April I really don’t know what happened but I’ve been working on some very good internal stuff and then here we are. I also do have some very cool stuff in the works for public consumption but that will have to wait. However I do have some other good stuff for you until I get it all going.

Stuff

Ask PFE Platforms- https://blogs.technet.com/b/askpfeplat/ This is a new team blog myself and a few other folks started. We post every Monday and sometimes more than that. As you can see I have a few posts I’ve written over there so I haven’t been completely worthless. If you aren’t already reading this you should it’s a great spot.

A collection of Slow Boot Slow Logon links

https://social.technet.microsoft.com/wiki/contents/articles/10130.root-causes-for-slow-boots-and-logons-sbsl.aspx

https://social.technet.microsoft.com/wiki/contents/articles/10128.tools-for-troubleshooting-slow-boots-and-slow-logons-sbsl.aspx

https://social.technet.microsoft.com/wiki/contents/articles/10123.troubleshooting-slow-operating-system-boot-times-and-slow-user-logons-sbsl.aspx

What I’ve Been Reading For Work

Windows Internals 6th ed Part 1- https://shop.oreilly.com/product/0790145305930.do I’m slowly but surely working my way through this. The more Xperf performance tracing I do the more I need to understand this. There should be some other xperf related posted on Ask Pfe Plat in the next month…ish.

Windows Sysinternals Admin Reference- https://shop.oreilly.com/product/0790145316974.do If you are not familiar with the Sysinternal tools, you should be. This is a great book.

What I Will Be Reading for Work

Windows NT File System Internals and Windows NT Device Driver Development- https://www.osronline.com/custom.cfm?name=index_fullframeset.cfm&pageURL=https://www.osronline.com/store/index.cfm Because I just cannot get enough of low level stuff. I just ordered these today.

Inside Windows Debugging- https://shop.oreilly.com/product/0790145335500.do This should be out next week sometime.

Programming Windows 6th ed- https://blogs.msdn.com/b/microsoft_press/archive/2012/04/21/mark-your-calendars-programming-windows-sixth-edition-is-coming.aspx This one actually is $10 if you get it before June 1st. How can you not pick this up?

What I’m Reading for Fun

Team of Rivals: The Political Genius of Abraham Lincoln - https://www.amazon.com/gp/product/0743270754/ref=s9_simh_gw_p14_d2_g14_i1?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=center-2&pf_rd_r=0K2RHAPJ6GZY5CGHEQ3Q&pf_rd_t=101&pf_rd_p=470938631&pf_rd_i=507846

I just finished reading this long long book. It’s extremely interesting and extremely long. Get yourself an edition that you don’t have to lug around.

Ready Player One- https://www.amazon.com/Ready-Player-One-A-Novel/dp/0307887448/ref=sr_1_1?s=books&ie=UTF8&qid=1336871267&sr=1-1

Set in the future but about today and the 80s current geek culture. I’m in.

DC The New 52- https://en.wikipedia.org/wiki/The_New_52 I’ve been really reading stuff in Comixology https://www.comixology.com/ since DC restarted all their series at issue #1. Warning its dangerously easy to just keep clicking buy on the next comic. I guess there are worst habits to have.

That’s all for now.

Mark “Why Aren’t There More Gambit Comics?” Morowczynski

I was doing a WDRAP and I ran across an interesting case that required me to do some digging that I thought I’d share this out in one source. When we took an xbootmgr trace which we do with all our WDRAPs we saw broken readyboot that was identical to what I talked about here. https://blogs.technet.com/b/markmoro/archive/2011/11/28/xbootmgr-part-2-readyboot-basics.aspx. No cache hits and a lot of cache misses. I explained what readyboot does and how it’s suppose to work and the hotfix that we can roll out to help remediate this in their environment. Then they asked me a great question, “Does this still matter if it is a solid state drive (SSD)?” The answer is no when an SSD is  detected the following things are not used, readyboot, readyboost, Superfetch and disk defragmentation. https://blogs.msdn.com/b/e7/archive/2009/05/05/support-and-q-a-for-solid-state-drives-and.aspx . The disk defragmentation service will be set to still run but no disks will be selected.

So now the next question is why is Windows not detecting that the hard drive is a SSD. The first thing we attempted was to update the firmware of the SSD as it was out of date. We rebooted and checked the Superfetch service which was still running meaning that the OS still hasn’t picked up the disk as a SSD. Ok clearly this was something else. After doing some research I learned the following. The Superfetch service looks at the following registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winsat\DiskScore every 30 minutes. If the regkey is 65 or greater, which means the disk has the performance of an SSD, it sets the Superfetch service to manual. This score is populated by the actual WinSAT Primary Hard Disk score. Our next step was to check the WinSAT score for this machine. This is what we found.

Ok so that explains it, the WinSAT has never run on this computer. When is this suppose to run? It runs initially on start up to determine Aero performance and the rest is suppose to fully populate when the machine is idle via a scheduled task. The schedule tasks is under Microsoft\Windows\Maintenance\WinSAT. We found that during the image process this tasks was being disabled so it never ran and thus, never detected the machine had an SSD. Once we kicked of that scheduled task, it rated the Hard Disk at 7.1 the reg key had a value of 71, we waited about 20 minutes and all services were stopped and set appropriately.  Case closed.

Mark “my hard disk score goes to 11” Morowczynski

It goes back to our old friend that we seem to always forget, the event log. You'll want to navigate to Microsoft-Windows-ReadyBoost/Operational and look for Event ID 1016 about 1 minute after boot. This is logged every time the bootplan is calculated. The result should be 0x0. If you're having the issue, the data will be 0x57. 

Ready broken

How do I fix this for all my machines? Well, so far in the cases I've seen were able to fix this with this https://support.microsoft.com/kb/2555428. Now to be CLEAR about this, System Restore Points is not the ONLY way this file can hit its 512 KB size restriction. Other things such as a heavily loaded boot process(lots of apps, drivers, registry, etc) can cause this to occur as well. After applying the update go ahead and reboot. You should be all set. To confirm you can look for the same Event ID above but with 0x0 value.

Ready fixed!

I want to send a thanks to my down under, right near the end of the world Australian PFE Roger Southgate, hotlanta's own PFE Jeff Stokes(whose blog you should also read https://blogs.technet.com/b/jeff_stokes/) , Eugene from the Product Group and Scott Ladewig  for his help with testing.

Mark "I want to go on vacation to where Roger lives" Morowczynski

Readyboot is boot acceleration technology that maintains an in-RAM cache used to service disk reads faster than a slower storage medium such as a disk drive. ReadyBoot reads (prefetches) data into the cache before it is requested. Prefetching optimizes disk access patterns by taking data locality and hard drive's performance characteristics into account. Read requests from system processes, services and user applications are then serviced out of the ReadyBoot RAM cache.

ReadyBoot plays a HUGE role on how fast Windows is able to start. If ReadyBoot is broken, the rest of your data can be skewed entirely. I check ReadyBoot first to see how it's looking before I move on to other areas.

Here is an example

There are 3 different colors I want to focus on. The first is blue are write requests to the disk. The second is black, these are misses which means a read requests was serviced from the disk instead of the ReadyBoot cache. This is ReadyBoot death. The third color is green. These are ReadyBoot hits which means the read request was satisfied from the ReadyBoot cache. Notice there is NO green in this picture. ReadyBoot is essentially broken in this trace. The boot performance reflects this as ReadyBoot is finishing after 130 seconds, there is still much more disk activity taking place after that.

So now we need to fix this. We run the following command.

xbootmgr -trace boot -prepsystem

This will cause the system to reboot 6 times and order the files in the best order for performance. Now let's look at the trace after this was run.

Notice all the green. ReadyBoot is done loading fully 70 seconds into the boot. A full 1 minute earlier then the previous trace which drastically cut down the boot time.. In a future post I'll talk about the other colors, red and orange.

Now I'm sure you are thinking, "How do I massively detect for this in my environment?" and "How do I massively fix this in my environment?" I don't have a good answer to those questions.....yet

Update: There is a better way, I spell it out here, https://blogs.technet.com/b/markmoro/archive/2011/12/07/slow-windows-7-boot-one-readyboot-fix-that-you-ll-want-to-apply-probably-anyways.aspx

Mark "ReadyBoot me some Great Lakes Christmas Ale" Morowczynski

Here is a quick post. A question I get semi frequently is "how do I stay up to date on the latest hotfixes for 'insert product name' ?" Great question. The best way I can think to do it is through RSS. Here is a listing of a ridiculous amount of Microsoft products and their RSS feeds. Happy KB hunting, 

https://support.microsoft.com/selectindex/?target=rss#top

Mark "clear eyes, full hearts" Morowczynski

If you haven't read it I recommend you stop what you are doing and read it! So now you see how important that is to manage. There is a new KB Hotfix I recommended highly reading through it and then going through the proper testing channels for your environment. 

MS KB 2618669

Some key highlights directly from the KB

Cause:

In certain rare circumstances, a domain controller may issue recurring requests for RIDs from the global RID pool every 30 seconds.

If repetitive requests for RID pool updates are unchecked for a significant time, the global RID pool may experience too much consumption, and in extreme cases, may experience complete exhaustion of the global RID pool.

Resolution:

To prevent too much consumption of the global RID pool, we recommend that you take the following actions:

• Install this hotfix on all existing Windows Server 2008 R2 domain controllers.
• Integrate the update into the Windows Server 2008 R2 installation media. It guarantees that future domain controllers also have this update.

Happy testing!

Mark "No Diving In the Shallow End Of The RID Pool" Morowczynski

Where my luggage got here a full day later then me. I'm here to deliver an Active Directory Risk Assessment Program or an ADRAP which does an exhaustive check of the health/risk of Active Directory. If you've never had one I suggest contacting your TAM it will be an eye opening experience. One area of focus is how is time configured for the AD Forest? I want to touch on a not always seen scenario quickly.  Hopefully everyone has read this article https://technet.microsoft.com/en-us/library/cc773013(WS.10).aspx,. To summarize the PDC in the root domain should get it's time from a reliable external NTP server. All DCs in the parent domain get their time from their PDC. The PDC in a child domain can get it's time from the PDC in the root domain or any other DC in the root domain. Any child DC can get their time from the PDC in their domain or any DC in the parent. Workstations get their time from any DC in their domain. Check out the nice diagram in the previous link.

We can check this by running the W32TM /monitor. I'll use my MSPaint skills to protect the innocent.

Let's talk about the stratum quickly. This is ripped directly out of the previous link, I told you to read it  "The degree to which a computer’s time is accurate is called a stratum. The most accurate time source on a network (such as a hardware clock) occupies the lowest stratum level, or stratum one. This accurate time source is called a reference clock. An NTP server that acquires its time directly from a reference clock occupies a stratum that is one level higher than that of the reference clock. Resources that acquire time from the NTP server are two steps away from the reference clock, and therefore occupy a stratum that is two higher than the most accurate time source, and so on. As a computer’s stratum number increases, the time on its system clock may become less accurate. Therefore, the stratum level of any computer is an indicator of how closely that computer is synchronized with the most accurate time source."

As we can see in our picture above the PDC has a stratum of 3 and the other DCs have a stratum of 4. The PDC is closer to the most accurate time source such as a hardware clock so it's stratum is closer to 1 than the other DCs. This exactly what we are seeing.

What if this scenario happens however, your root PDC is not configured to get from an external time source but is getting it's time from itself and advertising itself as stratum level 1. Your other DCs now have a stratum level of 2. You realize the error of your ways you reconfigure your root PDC and now his stratum level now becomes  4 like the picture above. How do the other DCs behave? How do the other clients behave? Do they accept this stratum change?  For those that say that will never happen, it happened to a customer last week. When the stratum number of a time server increases, clients will refuse to accept any new time samples from them. This is by design and in the NTP protocol. So now your clients can drift out of sync! The way around this is the time service would need to be restrated on all clients. This resets the state and the clients will then begin to work normally after. Thanks to Sarath Madakasira for this tip. That's all for now.

Mark "can deliver ADRAPs in jeans and a mouserat tshirt" Morowczynski

I spent this past week in Windows Desktop Risk Assesment Program (WDRAP) training which was actually pretty fantastic. If you are a Premier customer talk to your TAM about getting one scheduled. A large portion of ours was taught by a Senior PFE Yong Rhee. One of the areas I'm starting to pick up on is a tool called xPerf and xBootmgr. It's fantastic and I need to spend far greater time on it. One of the things we do is analyze a boot trace and look for improvements. I'm horrible at this today but I'm getting better everytime I look at them. Smash cut to my own laptop that for at times seems to be taking forever on boot up. Time to investigate.

1.) run xbootmgr -trace boot

Wait for my laptop to reboot and the traces to merge and let see what is going on. So after looking to see what time services start up and how long they take to start up something jumps out immediately.

That arrow pointing up is  to the service starting called sftlist.exe which is Microsoft's App-V. The arrow pointing down is to how long it's taking to start, that counter is in seconds. It's currently taking 195 seconds to start. While it's trying to start up, all the rest of the services to the right aren't able to which is part of the reason I'm having a long login delay. There must be some way to fix this. Perhaps this is a known issue?

Bam!

This seems to be describing the behavior I'm having. Let me apply the update and try the xbootmgr again and see if it gets any better.

I had to zoom in so far to see now how long it's taking that it cut off the time axis. It is now taking 14 seconds. I just saved 181 seconds off my boot time with that one update. Umm holler! There is still plenty of work that I can do on my machine to speed it up and I'll continue to post more as I dive deep down into the world of xPerf and xBootmgr.

Mark  "it can't rain all the time" Morowczynski

(This is what it should look like if you would just stop monkeying with it)

Ok so what does it look like if "someone" actually made a manual connection object?

(It's horrible looking)

Ok so the next thing we discussed was how do help customers fix this is the best way possible. The easiest way is pretty straight forward, right click the connection object, delete. Seems pretty simple. But there is a better way to do this. And this is what David and I talked about. What if that manual connection object was correct? The KCC will then see that it needs a connection and re-make one. So what. The problem is that when a new connection object is made a VVJoin has to be done https://technet.microsoft.com/en-us/library/cc758169(WS.10).aspx. This can be a semi-painful process. If you have to do it thats fine but why do it if you dont have to do it? So the solution you really want to do is make that manual connection object to be managed by the KCC.

To do this follow these steps

1.) Open ADSIEdit and go to the Configuration partition.

2.) Drill down to Sites, the site where the manual connection object is, Servers, the server where the manual connection object is created, NTDS Settings

3.) Right click on the manual connection object and go to properites

4.) Go to the Options attribute and change it from 0 to 1

5.) Either wait 15 minutes (that's how often the KCC runs) or run repadmin /kcc to manually kick it off

Just like that it will be managed by the KCC. If that is the best connection object it will continue to use it with no VVJoin, if not it will tear it down and then make the best connection object.

UPDATE: Fellow PFE Tom Moser blogs.technet.com/b/tommos let me know that if you have an RODC manual connection object the options attribute will need to be changed from decmial value 64 to 65. It will then show up as IS_GENERATED | RODC_TOPOLOGY 

Mark "Insert Your Favorite 2 Pac Lyric" Morowczynski 

I had a fellow TAM ask me that quesion the other day. They had a customer that wanted to run a Microsoft application in Hyper-V. They had no idea if it was fully supported. Now you can!

https://www.windowsservercatalog.com/svvp.aspx?svvppage=svvpwizard.htm

You are now able to select the product, the VM platform, and see what support you get!

Mark "I promise to write more content as soon as I get a hot minute" Morowczynski