Martin's Windows Server Blog

A few more of my posts to AskPFE Platforms...

Martin Lucas — Fri, 18 Oct 2013 00:00:00 GMT

For those that follow this blog, there are a few other posts I've written that are on the AskPFE Platforms page that I did not cross-post here. They are:

http://blogs.technet.com/b/askpfeplat/archive/2013/10/21/storage-spaces-how-to-configure-storage-tiers-with-windows-server-2012-r2.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/10/02/sweet-sixteen-a-milestone-for-windows-clustering.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/09/25/storage-spaces-understanding-expansion.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/09/23/using-powershell-to-gather-performance-data.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/03/29/mailbag-how-do-you-set-network-adapter-settings-with-powershell-in-windows-8-or-windows-server-2012.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/02/21/mailbag-should-i-use-the-same-network-adapters-for-all-interfaces-on-my-cluster.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/01/25/mailbag-should-servers-be-restarted-as-soon-as-possible-after-updates-or-okay-to-wait.aspx

Windows Server 2012: Does ReFS replace NTFS? When should I use it?

Martin Lucas — Tue, 01 Jan 2013 22:39:00 GMT

When I first heard of ReFS my immediate first question was whether ReFS as a file system was a direct replacement for NTFS. It didn’t take long before my customers began to ask me the same question. In learning about this new file system in Windows Server 2012, it became apparent fairly quickly that ReFS, while a new file system, is built differently. REFS stands for Resilient File System. NTFS has its place, and so does ReFS. While ReFS may appear to have some similarity to NTFS, it does not contain all the underlying NTFS features and scales efficiently to handle data sets far larger than NTFS.

The Resilient File System

Although ReFS inherited some of the NTFS code base initially, it is a different file system with different uses in mind. In fact, disk tools that work with the NTFS Master File Table (MFT) won’t be able to work with ReFS because ReFS has its own mechanism for keeping up with file metadata. ReFS is ideal for storing large amounts of data and can be leveraged for file shares. Applications that run locally on the server and rely on specific NTFS features may not work with ReFS. However, many may have no issues due to ReFS compatibility with many of the Win32 storage APIs. For example, Windows Deployment Services (WDS) explicitly requires NTFS because it relies on specific features in order to implement the RemoteInstall folder structure used for storing images. These are features that a conventional file server or data repository does not require.

CHKDSK isn’t applicable to ReFS. Yes…I did just indicate that there’s no need to run CHKDSK on a ReFS partition. Are you feeling that the tool you’ve wanted to avoid for so long is now something you might want to hold onto…just in case? A love-hate relationship perhaps? The counselor is in. It’s okay to have those feelings if you have them. The truth is that in terms of ReFS partitions, ReFS doesn’t need CHKDSK because repair functionality is built-into the file system. Repair, if needed, occurs on-the-fly. Yep…there’s no need for extra tools to go fix corruption like with other file systems. And for what it’s worth, Windows Server 2012 contains improvements for CHKDSK.

ReFS can use checksums to detect if data has changed since last written and is able to detect and recover from corruption quickly. In fact, when data is written to disk, it is written to a new location on disk rather than over the top of existing data. Once successfully written, the file system can free the space used by the old data stream. ReFS is able to recover from corruption within the file system rapidly without limiting availability of the volume. Further, ReFS may be used with clusters, Hyper-V, file shares, data archival, and many other uses.

Additional integrity data streams may be enabled if you have additional needs for data protection. By default, ReFS uses conventional streams which behave identically to NTFS data streams. However, don’t think that because this is so that file system metadata is conventional also. File system metadata is protected against corruption. If you want additional protection of data streams, you may enable Integrity Streams. When configured to do so, checksums are used against written data and updates are done using copy-on-write. You may enable Integrity Streams on particular folders, volumes, or even granularly on a per-file basis. When coupled with redundancy through Storage Spaces, it is default behavior for Integrity Streams to be enabled for the entire volume. Storage Spaces and ReFS complement each other. When coupled with a mirrored Storage Space, duplicate copies of data will automatically be leveraged by ReFS. With this configuration, if corruption were to be encountered, ReFS can immediately leverage redundant data within Storage Spaces to expediently address the issue. One other example of how Storage Spaces compliments ReFS would be how ReFS periodically scrubs file system data to look for differences in an event to combat bit flips that can occur over time due to data stored over a long period of time in the same location.

As a file system, ReFS is not only good for resiliency, it is great for maintaining extremely large amounts of data. With data integrity and recovery features built-into the file system, there is no need to wait for CHKDSK to run to fix corruption. What if you needed to store 500 billion gigabytes in one place?. But why stop there? I recently read that ReFS can handle up to 1 Yottabyte (YB). That wasn’t in my geek vocabulary until now and I’m not sure I’m going to remember that name next week. To give you an idea of how large a number that is, 1GB can be represented as 10⁹. 1YB is represented by 10²⁴. That is like 1 quadrillion GB. That seriously messes with my head. Good luck purchasing that much available hard drive space for your home lab in the basement. Imagine trying to power all that storage without shorting out the power for the whole neighborhood. Can you imagine how long even a modern efficient version of CHKDSK might run on a volume that size? ReFS answers the needs of data integrity and efficiency and is mainly intended for very large volumes on file servers. It is a file system that goes beyond capability of NTFS.

NTFS Improvements

NTFS remains the file system of choice for the operating system boot volume as well as any other general needs for data storage. CHKDSK remains the tool of choice for dealing with NTFS file system issues should they happen. CHKDSK in Windows Server 2012 contains improvements…so CHKDSK continues to evolve and get better. The average NTFS deployment currently is around 500GB. In many cases administrators were hesitant to go beyond that due to the time potentially required to run CHKDSK against a volume that size. The time required to run CHKDSK has not been predictable due to file system structure complexity differences from one volume to another as well as a variety of other factors. The average number mentioned above has increased over time because the efficiency of CHDKSK has evolved with each new Windows Server release for quite some time now. As the following TechNet reference indicates, you can safely deploy multi-terabyte volumes based on the improvements in CHKDSK and the existing capabilities of the NTFS file system. Windows Server 2012 builds upon the self-healing capabilities of Windows Server 2008 R2 NTFS. NTFS fixes the corruption it can on-the-fly and what can’t be addressed immediately can already be calculated in how it needs to be fixed to that when you choose to mitigate the issue, the time required is truly minimal compared to the CHKDSK of years past.

http://technet.microsoft.com/en-us/library/hh831536.aspx

Comparing ReFS and NTFS Features

Although ReFS is a different file system, there are similar features between ReFS and NTFS. The easiest way to compare those is to look at the feature list side by side. Consider the following feature sets:

NTFS Features

ReFS Features

Supports Case-sensitive filenames

Preserves Case of filenames

Supports Unicode in filenames

Preserves & Enforces ACL's

Supports file-based Compression

Supports Disk Quotas

Supports Sparse files

Supports Reparse Points

Supports Object Identifiers

Supports Encrypted File System

Supports Named Streams

Supports Transactions

Supports Hard Links

Supports Extended Attributes

Supports Open By FileID

Supports USN Journal

Supports Case-sensitive filenames

Preserves Case of filenames

Supports Unicode in filenames

Preserves & Enforces ACL's

Supports Sparse files

Supports Reparse Points

Supports Open By FileID

Supports USN Journal

How did I pull this information? There are charts available on the net…and people that know me can testify that I like charts. Yet, it is easy to obtain the available features of each file system yourself from the live file systems that you already use. If you don’t have a volume available that uses ReFS, it is easy to create a VHD file, attach it, and format as ReFS to use for your own testing. Once you have NTFS and ReFS file systems at your fingertips, use FSUTIL to gather the file system supported features. Then it’s a matter of comparing each list of features. FSUTIL syntax is very simple:

FSUTIL fsinfo volumeinfo driveletter

This simple command lists the capabilities provided by the file system in use on the supplied drive letter. Simply run this command against each type of file system and compare.

Here are a few screen shots of using Disk Management to create a VHD, initialize as GPT, format as ReFS, and using FSUTIL against the newly created ReFS volume.

For this example I used dynamic expansion. If I were really going to use this VHD for data I would not thin provision like in this example. I only want to inventory
ReFS features for this example.

Right-click on the new disk. Choose Initialize. Choose GPT. Right-click on the disk and format as ReFS.

Notice that key capabilities missing from ReFS as compared to NTFS are EFS encryption, quotas, and compression. BitLocker may be used for encrypting these volumes while EFS is not an option. Thus, when using ReFS, encryption doesn’t need to be part of the file system. BitLocker satisfies the need for encrypting data on the volume as it encrypts the contents of the entire disk. Quotas may be managed outside the file system rather than through the file system like NTFS. Not supporting Hard Links is a key reason why you wouldn’t use ReFS for a system disk; files in the system32 folder are really hard linked back into the WinSxS folder structure. You might think that ReFS would have data deduplication built-into the file system. The fact that it doesn’t may not prohibit other components or third-party solutions from interfacing with ReFS through the API set provided.

Command Line Directives for ReFS

For those that are script fanatics (you know who you are), I’ve written this section just for you. We all know that the need for scripting language happens all the time. Perhaps you’re just creating re-creatable virtual environments for one reason or another…and now you need to include the creation of some ReFS volumes. Below are some valuable commands including conventional and PowerShell examples.

Formatting a ReFS Volume

While this can be done from the UI quite easily by choosing ReFS as the file system in the drop down from the Format dialog, this is also easily done from the command line. Full format example below:

Command-Line

Format /fs:ReFS J:

PowerShell Equivalent

Format-Volume –DriveLetter J –FileSystem ReFS -Full

In fact, typical command line syntax and optional parameters apply. Therefore, if you want this to be a quick format, just append /q to the above command line. However, you have the option to enable Integrity Streams for the volume. Note the following command:

Command-Line

Format /fs:ReFS /q /i:enable J:

PowerShell Equivalent

Format-Volume –DriveLetter J –FileSystem ReFS –SetIntegrityStreams $true

The preceding command enables Integrity Streams on drive J: and performs a quick format. The /i option offers you the ability to enable|disable this feature for the volume. If you enable this option, all files created on the volume will be configured with integrity. You may turn this off for individual files or folders using the Integrity command. However, know that if a file is non-empty and has data streams created with Integrity Streams, you cannot disable the feature for that file. You cannot change the integrity status for a file once the file contains integrity data streams. You could copy the file to another partition, delete the original, and then copy it back without Integrity Streams.

Additional Thoughts

During the beta for Windows Server 2012, there were blog posts on the net that mentioned a tool called INTEGRITY.EXE. In the released version of Windows Server 2012 this tool does not exist. This is not a bug or mistake. Development provided PowerShell cmdlets to address configuration options for ReFS instead of providing another utility to keep up with that had narrow focus to begin with. I plan to construct a future post on storage cmdlets that will include how to adjust ReFS behavior using PowerShell. Also, just because a file system is self-healing doesn’t mean that backups should be forgotten. If a large chunk of blue ice falls from the sky and crushes your massive ReFS file system in one loud thunk…you’re more than likely going to need to restore the data. While that's my conservative side speaking, unexpected events do happen. It never hurts to be prepared for the unexpected or the unlikely in a datacenter.

Windows Server 2012: How to Import a Storage Pool on Another Server

Martin Lucas — Mon, 24 Dec 2012 06:01:00 GMT

Thanks for great feedback and for reading my prior AskPFEPlat post regarding Storage Spaces in Windows Server 2012. Based on feedback and questions following that post, it seemed like a great time to show how to import a storage pool on another server. Storage Spaces in Windows Server 2012 writes the configuration about the storage pool onto the disks themselves. Therefore, if disaster strikes and the server hardware requires replacement or a complete re-install – there is a relatively simple procedure involved to mount and access a previously created storage pool...perhaps on another server. Notice I said server. The implementation of Storage Spaces on Windows 8 doesn’t offer the same feature set that Windows Server 2012 does, so therefore you can only import a storage pool on the same OS version for which it was created.

Step By Step Example

1. Make the disks available to the server

The first step, which may have multiple sub-steps, is to physically attach and successfully make available all drives from the pre-existing storage pool. To be successful in this endeavor, you would need appropriate drivers, cables, firmware, etc. to make this work. Obviously if these are SAS drives and the destination server doesn’t have a SAS interface, this will be somewhat of a difficult task without the proper interface. Once you’ve completed this task, from Server Manager you should see something similar to the following:

2. Make the storage pool read-write.

As you can see from the prior screenshot, the storage pool (as well as the virtual disk) has a warning exclamation mark symbol to indicate a problem. By default, Windows Server 2012 will detect the foreign volume and mark it read-only. The indicator appears because of this condition. To proceed, right-click the storage pool and choose Set Read-Write Access…

Make it read-write.

Choose the server where the storage pool may be mounted read-write. If this is a clustered storage pool, multiple nodes may appear in the Choose a Server for Read-Write Access dialog.

3. Attach the virtual disk.

The activities in prior steps make the storage pool available. However, within the storage pool will be any virtual disks previously created using pool resources. Just like a VHD, these must be attached in order to be accessible. Therefore, the yellow exclamation warning sign indicates that the virtual disk is not currently attached. To proceed, right-click on each virtual disk and choose Attach Virtual Disk.

If all is successful, each Virtual Disk chosen will be attached.

4. Online the resulting logical disk(s).

As you may recall, Windows Server 2008 Enterprise and Datacenter editions, by default do not automatically mount every disk device seen (providing that the default SAN policy has not been changed.) Windows Server 2012 is no different. To use any of the virtual disks that belong to the imported storage pool, you must online them first.

In Server Manager under Volumes\Disks, select the disk, right-click, and choose online.

Notice at this point that the volume now appears and remains accessible.

Concluding Thoughts

Hopefully the above notes and screenshots illustrate how easy it is to import a storage pool when you have a need to do so, or as a proof of concept when learning about Storage Spaces. The example above was created using Hyper-V Virtual Machines, and using iSCSI targets (for example only, not production) made available to a second newly installed VM after the first VM was conveniently destroyed. You could also use individual VHD files as components for the storage pool members.

Windows Server 2012 iSCSI Target: Is it the same as the downloadable target for Windows Server 2008 R2?

Martin Lucas — Mon, 10 Dec 2012 12:53:00 GMT

Originally posted by Martin to AskPFEPlat

As mentioned in my prior post about Storage Spaces, Windows Server 2012 contains many new capabilities. The integrated iSCSI Target happens to be one of those. Many have used the downloadable iSCSI Target for Windows Server 2008 R2 from Microsoft since it was released for internet download. Prior to release for download the Microsoft iSCSI Target was available as a component of Windows Storage Server. Windows Server 2012 contains a native iSCSI Target and there’s no need to download one if you need to make storage available via iSCSI for other systems or virtual machines. No special licenses required. It is worthy to note that the included target is not identical to the downloadable target for Windows Server 2008 R2 – though they both provide block based storage.

Integrated iSCSI Target Differences

The new target contains improvements above and beyond the previous version. For instance, the new iSCSI Target would be of interest to you if you have any new 4k sector drives as it contains support for these newer devices. Development included some enhancements that improve performance rather comprehensively. Additionally, iSCSI virtual disks do not have to be hosted on dedicated physical disks. You may use Windows Server 2012 Storage Spaces to host them. However, incorporation of iSCSI LUNs into a Storage Spaces pool is not supported.

Management

There are a few different ways to manage iSCSI Targets under Windows Server 2012 which may be different than what some administrators may be familiar with previously. Those that have become adept with PowerShell will be happy to know there is yet more PowerShell support for iSCSI Targets. Increased PowerShell support is of no surprise since management interfaces are consistently backed by PowerShell cmdlets. Beyond PowerShell, the MMC snap-in method for managing iSCSI Targets is no longer the way to manage them. You may manage iSCSI Targets via the management interface integrated within Server Manager. Specifying initiator IQN identifier strings for defined targets can be hassle free using the new management interface in Windows Server 2012 with the option to query initiator identifiers of systems you choose. The new query option does not work with Windows versions prior to Windows 8 or Windows Server 2012. However, for newer systems that will access your iSCSI Target, the new query option removes the potential for mistakes. As a side note, the iSCSI Initiator remains a separate component with a management interface accessible from the Control Panel or Administrative Tools.

New in Windows Server 2012 is the Windows Storage Management API (SMAPI) that provides cmdlets for PowerShell and a new set of WMI classes for management of Windows storage. Having these additional interfaces opens up scripting possibilities as well as remote administration through a variety of methods for managing Windows storage. Also new to Windows Server 2012 is the Storage Management Provider (SMP) framework which also works with 3^rd party storage. The iSCSI Target included with Windows Server 2012 as released does not include a Storage Management Initiative – Specification (SMI-S) Provider. Therefore, the interfaces described above need to be used for management of targets. The following link suggests that with the introduction of SMAPI that the Virtual Disk Service (VDS) is being deprecated. Therefore, this is a time of transition to superior technology for storage management. Until then, VDS remains for compatiblity or for storage types that don't yet have a SMI-S Provider.

http://msdn.microsoft.com/en-us/library/windows/desktop/hh848071(v=vs.85).aspx

VHD Compatibility

If you’ve used one of Microsoft’s iSCSI Targets before, you know that when you create virtual disks for a given target, the underlying storage resides in a VHD file. In Windows Server 2012, these iSCSI Target VHD files are completely compatible with Hyper-V (and vice versa) for VHDs up to 2TB. Therefore, if you create a VHD using iSCSI Target that is within the constraints of Hyper-V, you can port that VHD directly over for use with a Hyper-V virtual machine. However, Hyper-V under Windows Server 2012 introduces a new virtual hard disk format known as VHDx. This format is not supported by the iSCSI Target and is not interchangeable.

Other Improvements

Prior releases of iSCSI Target would allow snapshots to be taken of one of the virtual disks with the ability to revert to a snapshot later. However, the methods used could require lengthy operations to complete the copy. Though the copy operation for snapshots could be slow, applying the snapshot to revert was fairly quick. Windows Server 2012’s implementation of snapshots for iSCSI is based on VSS, much more robust, and uses common VSS features. In addition to VSS support, in Windows Server 2012 the iSCSI Target integrates with Failover Clustering. Therefore, you may create highly available iSCSI targets.

In Conclusion

The built-in iSCSI Target within Windows Server 2012 is a nice addition to the OS. I've already found it to be a great asset for proof of concept, testing, demos, or helping to increase availabilty of storage in the basement lab for my home office...especially along with Storage Spaces. Creating targets for Windows Server 2012 Hyper-V clusters is a snap with the new management interface and better yet saves time.

More Information

Windows Server 2012 Storage Management

http://blogs.msdn.com/b/san/archive/2012/06/26/an-introduction-to-storage-management-in-windows-server-2012.aspx

Windows Server 2012 Storage Spaces

http://blogs.technet.com/b/askpfeplat/archive/2012/10/10/windows-server-2012-storage-spaces-is-it-for-you-could-be.aspx

What can be used to keep Active Directory data secure?

Martin Lucas — Mon, 10 Dec 2012 12:46:00 GMT

Originally posted by Martin to AskPFEPlat

With the increase in concern over data privacy and security, one of my customers recently asked about securing Active Directory data while stored on disk and what could be done in relation to network traffic. My first thoughts were BitLocker and IPSEC. However, there are a number of different considerations that factor in. Special thanks to Brent Whitlow for contributing to this post.

Protecting Active Directory Information on Disk

Active Directory data primarily resides in the NTDS.DIT file as well as accompanying log files. Therefore, you could use encryption technology like BitLocker to encrypt volumes that contain Active Directory data. BitLocker is ideal for protecting volumes holding Active Directory data because it works with features in the server hardware and firmware to provide secure operating system boot and drive encryption, even if protected disks are removed and someone attempts to read them on another system. While you may use Active Directory to store recovery information for BitLocker volumes, using an external data recovery agent for BitLocker would be more suitable when using BitLocker to protect Active Directory data. (If replicated Active Directory databases were to become corrupt as a whole or no other DCs were available to retrieve the key, storing the recovery key in Active Directory would be like storing the combination to a locked safe inside the safe and forgetting the combination.) Therefore, you would want to safely protect any BitLocker recovery keys so that they are accessible to appropriate personnel, secure, and not prone to compromise.

Encrypting File System (EFS) is another encryption technology but not good to use for encrypting Active Directory data due to a circular dependency at boot time. Active Directory requires EFS to unlock the database before it can respond to queries. Yet, in order for EFS to unlock the file, it needs Active Directory. EFS and BitLocker operate at completely different levels. Therefore, using EFS you can make Active Directory too secure by creating a deadlock situation between Active Directory and EFS.

Protecting data on a volume is important, but what about other forms of Active Directory data? Any backups containing Active Directory data must also be secured. When using BitLocker on a volume, backups of those volumes may not be encrypted. Further, physical security of backups remains paramount.

Any virtualized DCs in the environment? Access to stored virtual hard disk files need to be protected as it the equivalent of a physical hard drive of a physical domain controller. Virtualized DCs need the same amount of care and consideration if not more. It is relatively easy for someone with administrative access to the virtualization host to have access to the files on that host. Choose only reliable and trusted administrators to have administrative access to the virtualization hosts that contain virtualized DCs. Further, use of removable storage devices can be limited by Group Policy. As such, you may want to limit who can write to removable storage or limit these devices to read-only access.

Protecting Active Directory Data over the Network

Transmission of Active Directory data over the network may be protected quite easily using methods like IPSEC or 802.1x if these are already in use within the environment. LDAPS and LDAP become somewhat irrelevant over a secured connection when using IPSEC or 802.1x. However, LDAP is still used for things like client logon or related to Group Policy so this may be something to protect when not using encryption over the network. Standard permissions within Active Directory permit access to Authenticated Users. IPSEC or 802.1x would cover encryption of traffic for domain joined clients. Non-domain joined clients do not query personal data stored in Active Directory and would be protected by IPSEC or 802.1x once joined. Prior to or during the join process, personal data is not queried through LDAP and the user does not have access as an Authenticated User.

One additional item to check would be the use of the old Pre-Windows 2000 Compatible Access Group for older clients like NT4. Using this option definitely makes Active Directory less secure.

Additional Resources

Windows Server and Processor Cores… (Part 2): Windows Server 2012

Martin Lucas — Mon, 10 Dec 2012 12:40:00 GMT

Originally posted by Martin to AskPFEPlat

My last installment about how Windows Server handles processor cores discussed Windows Server 2008 and Windows Server 2008 R2. Windows Server 2012 is out and the editions available differ somewhat from prior Windows operating system releases. With the editions available, key differences are not regarding features as one might think but actually concern virtualization rights. Both editions support the same amounts of memory, physical processors, and logical processors. This post is for informational purposes only. Please consult the End User License Agreement (EULA) for your product and/or the link at the end of this post for more information.

Windows Server 2012: Standard VS Datacenter

These two editions contain all the same features. For example, previous Standard editions did not contain Failover Clustering. Therefore, to use that feature it required Enterprise or Datacenter. Enterprise edition for Windows Server 2012 does not exist. You will find that Failover Clustering as well as any other feature you can find in Windows Server 2012 Datacenter also exists with Standard edition. Virtualization and processor licensing are key differentiators.

Licenses and Physical Hardware

Client Access Licenses (CALs) remain required for access to Windows Server 2012. Therefore, CAL requirements have not changed. As in the past, CALs must be equivalent or higher in version as compared to the Windows Server version being accessed. Therefore, Windows Server 2012 requires Windows Server 2012 CALS. Licensing requirements for Remote Desktop Services (RDS) and Active Directory Rights Management Service (ADRMS) remain the same and still require appropriate CALs for access.

What is different is what the Windows Server license provides. A single license for either available edition provides use of up to two physical processors. If you have additional physical processors in the same server, simply assign additional licenses for Windows Server 2012 for the edition purchased as needed for the additional set of physical processors. When I say ‘assign’ another license, I might as well say allocate because this is for licensing compliance. Currently you can only register one license key per server. Just make certain you have enough licenses purchased to cover the number of physical processors and for any virtualization needs (discussed in the next section). Remember, each license provides for up to two additional physical processors. The number of cores available per processor is irrelevant. If you have an odd number of physical processors, you can’t purchase server licenses for individual physical processors; they cover up to two. For purposes of example, if you assign four Windows Server 2012 Standard edition licenses to the same server, that server is licensed for use of eight physical processors and as many cores per physical processor as the OS will support.

Licenses and Virtualization Rights

Standard server licenses provide the ability to host two virtual machines (VMs) on those same two physical processors that the license covers. The same license also provides virtualization rights. Using the above example with four Standard edition licenses, the server is licensed for use of eight physical processors, as many cores as the OS supports, and up to eight VMs. If you have a need to support ten VMs on the same four (physical) processor system, simply assign one more Standard edition license to support ten VMs even though the hardware may only have four physical processors. The number of licenses needed depends on the number of physical processors or VMs to be hosted – whichever number is higher.

A Datacenter license provides the ability to host an unlimited number of VMs on those same two physical processors. However, like a Standard edition license, you need to assign additional Datacenter edition licenses based on the number of physical processors needed. The number of cores per physical processor remains irrelevant other than staying within the bounds of the OS and, of course, what the hardware itself will support. My take on this is that if you are going to be hosting a serious number of VMs on a host with lots of processors and cores, Datacenter is likely the most cost effective path.

Standard Edition to Datacenter Edition?

Standard Edition may fit some situations. However, Datacenter is obviously more flexible in terms of virtualization rights and is likely a better fit for highly virtualized environments. While with either edition you may assign more licenses, those licenses have to be for the same edition and may not be intermixed. So what can you do if you go with Standard and find out that the future of the server requires more licensing capacity? You could continue to purchase and assign additional Standard edition licenses as applicable. If you have Software Assurance benefits, you may be able to purchase a Step-up to migrate to a Datacenter license for that server.

How Many Processors Supported?

Windows Server 2012 supports up to 64 physical processors with an upper limit of 640 logical processors. Hyper-V enabled systems have an upper limit of 320 logical processors. Physical cores remain a preferred option in lieu of logical processors from Simultaneous Multithreading (SMT) when physical cores are available.

Concluding Thoughts

My thoughts behind writing this post were to help communicate a significant change that affects server deployment planning. The cost of licenses for Standard and Datacenter editions are different to account for the licensing changes. The price per instance appears to have been reduced such that equivalent instances for Windows Server 2012 may be less than Windows Server 2008 R2 Enterprise. Again, this post is not a definitive source for what a license covers, what the terms are, or what all the different options are that are available. A great source for frequently asked questions and other links to related information, please consult the following Windows Server 2012 Licensing and Pricing FAQ document:

http://download.microsoft.com/download/4/D/B/4DB352D1-C610-466A-9AAF-EEF4F4CFFF27/WS2012_Licensing-Pricing_FAQ.pdf

Visit the AskPFEPlat blog...

Martin Lucas — Mon, 10 Dec 2012 12:04:01 GMT

Wanted to take a minute to mention the PFE Blog AskPFEPlat that I've been known to post to once in a while. There you'll find a variety of great topics with great technical information. Any future posts I make to AskPfePlat will also cross-post here. I will continue to post material to Martin's Windows Server Blog that fit here but may not fit in the larger forum (Debug hints and tricks would be just one example.)

Happy Holidays!

Martin

Clustering: What exactly is a File Share Witness and when should I use one?

Martin Lucas — Wed, 27 Jun 2012 06:10:00 GMT

Customers ask from time to time: “What is a File Share Witness (FSW)?” Sometimes they’ve worked with prior versions of clustering and don’t know what a FSW is, or that the option exists. The next question asked is usually: “When should we use one?” Before going into that, I’ll review some subtle differences between legacy cluster quorum options and what options are available from Microsoft today for a Failover Cluster.

Legacy Cluster Quorum Options

Quorum may be defined as the number of members of a group that must be present to conduct business. For a legacy (Windows NT/2000) two-node cluster that lost all communications and became partitioned, whichever node could maintain reservation of the quorum disk first would survive and own it. The quorum disk was a tie-breaker for systems that could not communicate as well as an additional storage location for cluster configuration data. One downside to this model was that if the quorum disk failed, so did the cluster. A legacy two node cluster could not function without it. So if just the disk failed but both nodes remained, the cluster would cease to function. Therefore, the quorum disk was very important for legacy clusters. With Windows Server 2003 clusters with more than two nodes, Majority Node Set was another quorum option. A MNS cluster can only run when the majority of cluster nodes are available. This model is typically not chosen for two node clusters because with this model you have to have two nodes minimum ((2 / 2) +1 = 2)…to maintain majority. As of 2003 SP1, an option was added to allow use of a File Share Witness to add an additional vote so that in the same example above, a two node cluster could function with the loss of one node.

Quorum Options for Modern Clusters

With Windows Server 2008 and later clusters, if all network communication becomes severed between nodes, and quorum must be established, votes are counted. (The exception to this statement would be the disk only quorum model which is similar to the quorum model in legacy clusters where the only vote that is counted is the disk. This option is not recommended, and is rarely chosen.) By default, each node gets a vote and if configured, a single witness may be counted as a vote. A witness may be either a Witness Disk or File Share Witness (FSW). However, you cannot use both. Half of all possible votes + 1 must exist for there to be quorum. Therefore, for an even number of nodes you would want to have a FSW or witness disk. This means that the disk or FSW can cease to exist and as long as enough nodes are online then the cluster may still function.

The following TechNet link is a great reference for 2008 and 2008 R2 quorum options:

http://technet.microsoft.com/en-us/library/cc770620(v=WS.10).aspx

From that link, the following table describes when you would use an alternate method based on even or odd number of nodes.

Description of cluster	Quorum recommendation
Odd number of nodes	Node Majority
Even number of nodes (but not a multi-site cluster)	Node and Disk Majority
Even number of nodes, multi-site cluster	Node and File Share Majority
Even number of nodes, no shared storage	Node and File Share Majority

What is a FSW and how does it differ from a disk?

All this discussion about votes is a perfect segue into what a File Share Witness (FSW) is and how it differs from a witness disk. A FSW is simply a file share that you may create on a completely separate server from the cluster to act like a disk for tie-breaker scenarios when quorum needs to be established. The share could reside on a file server, domain controller, or even a completely different cluster. A witness share needs to be available for a single connection, and available for all nodes of the cluster to be able to connect to – if you are using the FSW option for quorum. The purpose of the FSW is to have something else that can count as a vote in situations where the number of configured nodes isn’t quite enough for determining quorum. A FSW is more likely to be used in multi-site clusters or where there is no common storage. A FSW does not store cluster configuration data like a disk. It does, however, contain information about which version of the cluster configuration database is most recent. Other than that, the FSW is just a share. Resources cannot fail to it, nor can the share act as a communications hub or alternate brain to make decisions in the event cluster nodes cannot communicate.

Remember the old capture the flag game you might have played at summer camp? Typically each team had a flag and had to protect it from capture by the opposing team. However, a variation on that game is where there is only one flag located at an alternate location in the woods and two teams try to find it. When the flag is captured by one team, that team wins. A FSW is somewhat like the flag at an alternate location where a team of surviving nodes that can obtain it are able to reach quorum and other nodes that cannot drop out of the active cluster.

A Witness Disk is similar to a FSW except rather than being a file share somewhere on your network like a FSW, it is an actual disk provided by disk storage that is common to the cluster. Nodes arbitrate for exclusive access to the disk and the disk is capable of storing cluster configuration data. A witness disk counts as a vote.

By this point you may have a good understanding of what a FSW is, when it might be used, what it is, and what it isn’t. Now let’s look at a couple of make-believe cluster configurations that each use a FSW that are similar but quite different.

This is a 4 node cluster that is split between two sites with a FSW in Site C. In this configuration there are 5 possible votes. Bidirectional communication is possible on the Public and Private networks. Site A and Site B have a link to Site C to connect to the FSW, but neither Site A or Site B may connect with each other through Site C’s network. If one of the bi-directional networks fails, there remains one network that may be used for cluster communications for the nodes to determine connectivity and decide the best recovery method. If both bi-directional networks fail, then this cluster is partitioned into two sets of two nodes and the FSW must be leveraged to determine which set of nodes survives. The first set of nodes to successfully gain access to the FSW will survive. IF the network to Site C were a complete network allowing communication between Site A and Site B as well, then there would be an alternate communication path for the cluster to determine the best course of action…and this configuration would be that much better.

This variation is very similar to the first example and is something that customers have been known to implement. In this case, both bi-directional networks are actually VLANs that go through the same network connection. Therefore, the two separate networks have the vulnerability of a single network. Even if the Leased Net were redundant, that piece of this puzzle remains provided by the same network provider or could be cut by a backhoe somewhere between. So there exists a possibility the Leased Net segment could go down or become unresponsive. The validation process will warn if there is only a single network but it has no insight into what the underlying network actually is between the two sites. An extra network or bi-directional communication link through Site C would once again be an improvement.

Without a FSW in these two configurations, there would be no way to break the tie during communication loss as there are two sets of two nodes otherwise.

Let’s take a non-technical example. Imagine you have 4 people on a camping trip. They park the truck at the camp parking lot and split into two groups of two. Each group hikes 5 miles in opposite directions from the truck they arrived in but at most, the groups are 5 miles from each other. They each make camp in a triangular setup like the second diagram above. Each group has a cell phone to communicate (with voice and text capability) and a heater. The truck contains a spare heater with a partial tank of fuel. Late at night, the heater at each camp fails. One runs out of fuel. The other experiences a mechanical problem. One or both groups are beyond cell range and cannot communicate. The first person that hikes back to the truck to get the spare heater has heat for their tent at their campsite. It doesn’t matter if both camps can hike to the truck or can even see the truck directly from their camp. There is only one spare heater. Since there is no communication available, they can’t call or txt message each other. The camp that ran out of fuel ends up getting the spare heater and almost enough fuel to make it through the night. The other camp wastes energy of one person hiking to the truck and back to find there is no spare heater and ultimately freezes with no working heater and extra fuel they can’t use. Ideally, if the camps could communicate by other means, they would know to meet at the truck, swap the broken heater for the spare and divide up all the fuel so that both camps could have heat, or decide to leave and get coffee on the way home. With only one real communication path, the best decision for all 4 people could not be made.

The cluster configuration is similar. With all communication severed, the cluster has to decide what to do based on the information available which may be limited. The FSW in the cluster example is capable of breaking the tie of two votes against two votes. However, the cluster is not able to help discern which site is actually the best one to continue because of other conditions.

How do I know if I’ve chosen the best quorum model?

For a cluster with an odd number of nodes, Node Majority is the typical model chosen. However, with an even number of nodes it makes sense to have a witness resource as a vote. The validation process for clusters in Windows Server 2008 R2 can validate the cluster against best practices and suggest a different quorum model if the current selection is not the best option.

VSS Timeouts during Backup? Check Fragmentation!

Martin Lucas — Wed, 13 Jun 2012 11:43:30 GMT

Volume Snapshot Services (VSS) was and remains a good addition to the Windows OS. Without going into a lot of detail, this technology coordinates various components to ensure stable point-in-time backups even while applications may be running. Typically, this occurs successfully with only minimal interruption to the application. On occasion there can be problems taking backups of systems that result from configuration issues, missing fixes, to third-party providers and writers that need updates. VSS really does a lot during the course of a normal backup. There are a couple of points during the process that actions must be timely. Failure of components to complete I/O within the allotted time results in timeout. Sometimes something as simple as file system fragmentation can be the reason behind such timeouts.

A really good link describing all the things that happen during a VSS backup may be found using the following link.

http://msdn.microsoft.com/en-us/library/aa384589(v=VS.85).aspx

Using the above link you will find a chart of VSS activities. Common places for trouble in this process are the shaded green and red areas. Consider the below excerpt from the chart:

Each of these two highlighted areas denotes the finite amount of time for which operations must complete to avoid timeout. One particular configuration issue that can cause timeouts and backup failures at this juncture is file system fragmentation. Sounds pretty trivial right? Sure. For most systems and most environments a defragmentation plan may exist to prevent issues and to ensure optimum performance. In fact, many clients have a scheduled task to defrag the system drive automatically. In the case where the volume isn’t defragmented regularly and there are no other combined issues, running defrag.exe may resolve the condition quickly to allow successful backups. However, if you have a file system that was upgraded from an older operating system – or by chance was formatted using an older deployment tool – it is possible that the file system may be using a smaller cluster size and may be more prone to ragmentation than the average system that uses a default cluster size or larger. It is also possible that systems like this may have other contributing factors that may prevent defrag.exe from resolving the issue of fragmentation preventing successful backups. It is recommended that systems use the default cluster size or larger for best performance.

Background on Converted Volumes and Small Cluster Size

If you find that a system volume has a small cluster size of 512 bytes (not to be confused with the sector size which is commonly 512 bytes), this file system might have been converted to NTFS at one point in the past by CONVERT.EXE. FAT could be the original file system later converted to NTFS. It is known that when converting to NTFS with CONVERT.EXE, the cluster size does not change from the original file system. Perhaps this system was upgraded to Windows Server 2003 or may have been formatted with some other type of deployment utility that used a small cluster size. When a volume is formatted freshly with NTFS, the file system uses a default cluster size of 4K providing the volume is less than 16TB. Volumes with a small cluster size are more likely to achieve significant levels of fragmentation faster than a volume of same size using a default or larger cluster size. Therefore, volumes with small cluster sizes may require more frequent defragmentation.

A second characteristic that may contribute to fragmentation is lack of free space. As a volume gets closer to being full, the likelihood of fragmentation increases. Combine this characteristic with a small cluster size and the likelihood of a highly fragmented volume increases dramatically.

A third contributing characteristic would be if the volume contains a page file configured to allow dynamic expansion as needed. If the page file experiences frequent extension on a volume that is already highly fragmented, the page file may become highly fragmented as well. A highly fragmented page file not only impacts performance but also may impede the ability to defragment the volume…especially if the volume is more than 60%-70% full.

The additional head and platter movements required because of the degree of fragmentation can easily result in VSS timeout because I/O operations have to be split into multiple I/O requests to retrieve data. Depending on the degree of fragmentation coupled with contributing factors may present a situation that defag.exe can’t easily resolve in order to complete a successful backup.

Mitigation of a Highly Fragmented Volume that Won’t Defragment

I recently consulted on a case where the page file itself contained over 500,000 fragments on a volume that was fragmented. For this volume, defragmentation activities could not get fragmentation on the volume below 20% and the delays to VSS consistently prevented successful backups. To resolve this condition in order to conduct a successful backup, the page file had to be removed from the system volume and located elsewhere, a restart performed, and defrag allowed to run a few times. Then after successful defrag and backup, we set the page file on the system volume to be a fixed size and sufficient for the system. The restart in this sequence of events is important because some file system space may remain locked after setting the page file on the system volume to zero. In fact, it might be better to approach this with a defragmentation utility being run from WinPE or safe mode so that the list of potential obstacles is even shorter.

The problem experienced on this system was one of cyclical nature contributed to by small cluster size, reduced amount of free space, and a dynamically expanding page file. Running defrag on a more frequent basis, removing page file issues, and keeping sufficient free space can prevent this type of situation from recurrence. Removing any one of the key issues would have had a positive impact on preventing the situation. However, the potential for fragmentation and VSS issues continues due to the small cluster size in use.

Typically for a system disk, the 4K default cluster size is adequate. For a volume to hold only databases you would likely use a much larger cluster size for better efficiency as recommended by the application. On any systems like this one that remain in production, it is advisable to monitor and address fragmentation on a regular basis to avoid issues. It may be desirable to negotiate down time to reformat the system volume with a larger cluster size and restore a known good backup to alleviate potential recurrence of issues and reduce the extra administrative overhead for a system using a small cluster size for the file system.

Determining the Cluster Size

It may be a good proactive step to check other 2003 servers for 512 byte cluster sizes on the system partition in an effort to increase awareness of volume and page file fragmentation and to adjust practices as needed. An easy command to display cluster size is as follows:

fsutil fsinfo ntfsinfo c:

Sample output from a 512 byte cluster size volume:

Bytes Per Sector : 512

Bytes Per Cluster: 512 <

Sample output from a 4096 byte (4K) cluster size volume:

Bytes Per Cluster : 4096 <

Additional Considerations

If using volumes with a small cluster size and using Shadow Copies for Shared Folders to have previous versions of files available, the number of changes caused by defragmentation on a highly fragmented volume can also cause shadow copies containing previous versions of files to be deleted. Therefore, for Shadow Copies for Shared Folders, it is a best practice to use 16K clusters or larger.

For more information about NTFS, please consult the following TechNet document:

http://technet.microsoft.com/en-us/library/cc767961.aspx

Debug: Why isn’t this IRP being serviced on a hung server?

Martin Lucas — Wed, 04 Apr 2012 12:24:00 GMT

Have you ever been following an I/O request in a crash dump that leads to another I/O Request Packet (IRP) – to then be at a dead end and not know why the driver isn’t servicing the IRP? If so, you are not alone. It could be that this transaction involves a work item and system worker threads. !analyze –v typically doesn’t go far enough to illuminate an issue like this and simply looking at locks and IRPs may not be enough. A deadlocked system may, in fact, look idle at first glance within the debugger.

Many times drivers and other kernel components create their own threads to process workloads. The kernel offers the ability for drivers and other kernel components to create work items to be queued and processed by a pool of system worker threads. Worker threads are re-used over and over and will show idle if there are no work items remaining in the queue. These work items can come from drivers or from other kernel threads (for example, at DPC level) where the thread cannot perform an action because of the current Interrupt Request Level (IRQL) of the processor. Such requests may be handled by worker threads that are able to execute later or by another processor.

When a thread creates a work item, it calls a kernel API (there are a few to choose from) to queue the work item. If you’re looking at a thread stack that has created the work item already, you will not see the API call used but instead may see calls that lead up to a wait for object call. Often what the thread is waiting on at this point is an event that signals when the work item completes. For more information on worker threads or work items, consult “Windows Internals” by Mark Russinovich and David Solomon.

Why did I bring up work items like that? Worker threads and queued work items can be very important things to check when debugging a hung system...and if you're used to debugging hang dumps that don't involve them -- it is easy to forget to look at them when the cause of a hang doesn't make sense. Debugging of crash or hang dumps is an acquired skill and it often helps to have good examples to refer back to. The following annotated debug session is a classic example of a deadlocked system that appears to be blocked on a Windows Server driver but in fact is blocked by something else entirely. The crash dump was manually induced on a hung system. Dumps like these can take many hours of analysis to prove or disprove involvement of components. Much of the output has been truncated appropriately and formatted for readability because some of these commands produce excessive amounts of information. Special thanks to the Debug Ninja for assisting with this one!

Using !locks reveals a lock held exclusively with many threads waiting on it.

0: kd> !locks

**** DUMP OF ALL RESOURCE OBJECTS ****

KD: Scanning for held locks...

Resource @ 0xfffffa9006bf5150 Exclusively owned

Contention Count = 1892

NumberOfExclusiveWaiters = 33

Threads: fffffa900b7915d0-01<*>

Threads Waiting On Exclusive Access:

fffffa900e50f060 fffffa900e7fe060 fffffa900c86f060 fffffa900eaa44f0

fffffa900edaebb0 fffffa900ebf4bb0 fffffa900d5e5740 fffffa900f3a3790

fffffa90065b58e0 fffffa900ae79720 fffffa900eddabb0 fffffa900d5de040

fffffa90065b4040 fffffa900b371280 fffffa90065c2720 fffffa900ae79bb0

fffffa900ec4dbb0 fffffa900c9b6bb0 fffffa90065c2290 fffffa90065b4720

fffffa900cf9cbb0 fffffa900d5c0bb0 fffffa90065b4bb0 fffffa90065b3bb0

fffffa900b668040 fffffa90065b3040 fffffa90065c2bb0 fffffa900f4a2bb0

fffffa900f0e1bb0 fffffa900edcdbb0 fffffa90065b3720 fffffa900cc5abb0

fffffa900ddc6060

Looking at thread fffffa900b7915d0 that owns this lock, it can be seen that the last driver on the stack before the thread went into a wait is fltmgr.sys which is a Windows driver.

0: kd> !thread fffffa900b7915d0

THREAD fffffa900b7915d0 Cid 1fa0.1d20 […] WAIT:(Executive) KernelMode Non-Alertable

fffffa600358bc98 NotificationEvent

IRP List:

fffffa900d9cd8b0: (0006,03a0) Flags:00000884 Mdl: 00000000

fffffa900ea51960:(0006,03a0) Flags: 00000884 Mdl:00000000

Not impersonating

DeviceMap fffff88011eaa630

Owning Process fffffa900eef5480 Image: cmd.exe

Attached Process N/A Image: N/A

Wait Start TickCount 81612 Ticks: 31801 (0:00:08:16.098)

Context Switch Count 206 IdealProcessor:1

UserTime 00:00:00.000

KernelTime 00:00:00.046

Win32 Start Address 0x000000004ab67370

Stack Init fffffa600358edb0 Current fffffa600358b9b0

Base fffffa600358f000 Limit fffffa6003589000 Call 0

Priority 14 BasePriority 8 PriorityDecrement 5 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

fffffa60`0358b9f0 fffff800`01a666fa nt!KiSwapContext+0x7f

fffffa60`0358bb30 fffff800`01a5b38b nt!KiSwapThread+0x13a

fffffa60`0358bba0 fffffa60`00d70142 nt!KeWaitForSingleObject+0x2cb

fffffa60`0358bc30 fffffa60`00d68fa4 fltmgr!FltpPostSyncOperation+0xc2

fffffa60`0358bce0 fffffa60`00d62cdb fltmgr!FltpGetNormalizedFileName+0x69d4

fffffa60`0358bd10 fffffa60`00d46979 fltmgr!FltpCreateFileNameInformation+0xab

fffffa60`0358bd40 fffffa60`00d51334 fltmgr!FltpGetFileNameInformation+0x3b9

fffffa60`0358bdd0 fffffa60`0b0cb9e0 fltmgr!FltGetFileNameInformation+0x174

fffffa60`0358be60 fffffa60`0b0cbea1 AnotherDriver+0x89e0

fffffa60`0358bea0 fffffa60`00d44eca AnotherDriver+0x8ea1

fffffa60`0358c060 fffffa60`00d43fce fltmgr!FltpPerformPostCallbacks+0x30b

fffffa60`0358c130 fffffa60`00d6026c fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x34e

fffffa60`0358c1a0 fffffa60`00e389bc fltmgr!FltpCreate+0x25d

fffffa60`0358c250 fffffa60`00e46a82 3rdPartyDrv+0x2e9bc

fffffa60`0358c290 fffffa60`00e4668f 3rdPartyDrv+0x3ca82

fffffa60`0358c330 fffffa60`00fc499f 3rdPartyDrv+0x3c68f

fffffa60`0358c3a0 fffff800`01cd29d9 3rdPartyDrv+0x1ba99f

fffffa60`0358c3f0 fffff800`01ccaed2 nt!IopParseDevice+0x5f9

fffffa60`0358c590 fffff800`01ccba65 nt!ObpLookupObjectName+0x593

fffffa60`0358c6a0 fffff800`01cd1047 nt!ObOpenObjectByName+0x2f5

fffffa60`0358c770 fffff800`01cdaed8 nt!IopCreateFile+0x287

fffffa60`0358c810 fffff800`01a63f33 nt!NtCreateFile+0x78

fffffa60`0358c8a0 fffff800`01a64440 nt!KiSystemServiceCopyEnd+0x13

In this thread, you can see that it is waiting on Executive activity, denoted by WAIT: (Executive). In fact, this thread has been waiting for over 8 minutes (see Ticks: 31801 (0:00:08:16.098)). Context has been swapped as noted with nt!KiSwapContext and it is waiting on notification event fffffa600358bc98. The event is tied to what this thread is waiting on and when signaled this thread may be scheduled to resume execution. However, we need to see what this thread is waiting on because so many threads are unable to execute pending release of the lock this thread owns.

At this point one could look at the IRPs that !thread shows. However, if you just look at the IRPs and stop there, you’ll have the wrong conclusion.

0: kd> !irp fffffa900d9cd8b0

Irp is active with 9 stacks 8 is current (= 0xfffffa900d9cdb78)

No Mdl: No System Buffer: Thread fffffa900b7915d0: Irp stack trace.

cmd flg cl Device File Completion-Context

[ 0, 0] 0 0 00000000 00000000 00000000-00000000