In working with customers building migration plans for Windows Server 'X' to Windows Server 2016, one of the new things to be aware of is that you can no longer "move" between a "Core" install and a "Full"/"Desktop Experience" install.  Even further is that Core now only supports a subset of features of "Desktop Experience" version.  This is identified on Technet where supported workloads for Core are documented.  However, I found that that said detail was not enough to interrogate the environment to determine which servers could vs. could not be upgraded to Core (preferred).

https://technet.microsoft.com/en-us/windows-server-docs/get-started/getting-started-with-server-core

Discovery

I wrote a PowerShell script that exported the list of Windows Server 2016 features from both Core and Desktop Experience.  The script is attached to the blog entry should you be curious.  It requires an online Desktop Experience and Core version, as well as the "WIM" for the install of each.  I know there are more elegant ways to do this:

• (Invoke-command for the Get-WindowsOptionalFeature) so all that is needed are the online installs.
• All of this could be done with the Get-WindowsOptionalFeature, but that gave me a raw list with none of the parent/child relationships that Get-WindowsFeature provided so it was a little rough to understand.
  

However, this got the job done and was not worth a re-write.

Script:  Compare Windows Server 2016 Desktop Experience and Core.ps1

Results

Here is a table of the Windows Features missing from core (see full spreadsheet Features Missing from Windows Server 2016 Core, includes insight into subfeature for which there is partial support of the top level feature):

NOTE:  You will notice that there is a column about whether or not this is an RSAT SubFeature.  For functional purposes we do not care when upgrading whether or not someone has installed administration tools that are dependent on the GUI.  Since we don’t want admins logging on to the boxes to do administration.

Building on top of this information

I used this information in System Center Configuration Manager to create two collections, one of systems that can be upgraded to Core and one of systems that must be upgraded to Desktop Experience.

Here are the respective SCCM Collection Queries (note that there may be better/performing ways to write these):

• Servers that can not be upgraded to core based on features installed
  select distinct SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_SERVER_FEATURE on SMS_G_System_SERVER_FEATURE.ResourceID = SMS_R_System.ResourceId inner join SMS_G_System_SYSTEM on SMS_G_System_SYSTEM.ResourceID = SMS_R_System.ResourceId where SMS_G_System_SERVER_FEATURE.Name in ("Fax Server","MultiPoint Services","Network Policy and Access Services","Distributed Scan Server","Internet Printing","Remote Desktop Gateway","Remote Desktop Session Host","Remote Desktop Web Access","IIS Management Console","IIS 6 Management Console","Windows Deployment Services","Deployment Server","Transport Server","IIS Server Extension","BitLocker Network Unlock","Direct Play","Internet Printing Client","LPR Port Monitor","Multicasting Support","RAS Connection Manager Administration Kit (CMAK)","Remote Assistance","Simple TCP/IP Services","SMTP Server","TFTP Client","Windows Biometric Framework","GUI for Windows Defender","Windows Identity Foundation 3.5","Windows PowerShell ISE","Windows Search Service","Windows TIFF IFilter","Wireless LAN Service","XPS Viewer") and SMS_G_System_SYSTEM.SystemRole = "Server"
• Servers that can be upgraded to core based on features installed (excluding RSAT features):
  select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_SYSTEM on SMS_G_System_SYSTEM.ResourceId = SMS_R_System.ResourceId where SMS_G_System_SYSTEM.SystemRole = "Server"  AND SMS_R_System.Name NOT IN (select distinct SMS_R_System.Name from  SMS_R_System inner join SMS_G_System_SERVER_FEATURE on SMS_G_System_SERVER_FEATURE.ResourceID = SMS_R_System.ResourceId inner join SMS_G_System_SYSTEM on SMS_G_System_SYSTEM.ResourceID = SMS_R_System.ResourceId where SMS_G_System_SERVER_FEATURE.Name in ("Fax Server","MultiPoint Services","Network Policy and Access Services","Distributed Scan Server","Internet Printing","Remote Desktop Gateway","Remote Desktop Session Host","Remote Desktop Web Access","IIS Management Console","IIS 6 Management Console","Windows Deployment Services","Deployment Server","Transport Server","IIS Server Extension","BitLocker Network Unlock","Direct Play","Internet Printing Client","LPR Port Monitor","Multicasting Support","RAS Connection Manager Administration Kit (CMAK)","Remote Assistance","Simple TCP/IP Services","SMTP Server","TFTP Client","Windows Biometric Framework","GUI for Windows Defender","Windows Identity Foundation 3.5","Windows PowerShell ISE","Windows Search Service","Windows TIFF IFilter","Wireless LAN Service","XPS Viewer") and SMS_G_System_SYSTEM.SystemRole = "Server")

(Update) PowerShell command string to determine which roles prevent a system from being core:

Get-WindowsFeature|Where-Object {$_.Installed -and ($_.DisplayName -EQ “Fax Server" -or $_.DisplayName -EQ "MultiPoint Services" -or $_.DisplayName -EQ "Network Policy and Access Services" -or $_.DisplayName -EQ "Distributed Scan Server" -or $_.DisplayName -EQ "Internet Printing" -or $_.DisplayName -EQ "Remote Desktop Gateway" -or $_.DisplayName -EQ "Remote Desktop Session Host" -or $_.DisplayName -EQ "Remote Desktop Web Access" -or $_.DisplayName -EQ "IIS Management Console" -or $_.DisplayName -EQ "IIS 6 Management Console" -or $_.DisplayName -EQ "Windows Deployment Services" -or $_.DisplayName -EQ "Deployment Server" -or $_.DisplayName -EQ "Transport Server" -or $_.DisplayName -EQ "IIS Server Extension" -or $_.DisplayName -EQ "BitLocker Network Unlock" -or $_.DisplayName -EQ "Direct Play" -or $_.DisplayName -EQ "Internet Printing Client" -or $_.DisplayName -EQ "LPR Port Monitor" -or $_.DisplayName -EQ "Multicasting Support" -or $_.DisplayName -EQ "Multicasting Support" -or $_.DisplayName -EQ "RAS Connection Manager Administration Kit (CMAK)" -or $_.DisplayName -EQ "Remote Assistance" -or $_.DisplayName -EQ "Simple TCP/IP Services" -or $_.DisplayName -EQ "SMTP Server" -or $_.DisplayName -EQ "TFTP Client" -or $_.DisplayName -EQ "Windows Biometric Framework" -or $_.DisplayName -EQ "GUI for Windows Defender" -or $_.DisplayName -EQ "Windows Identity Foundation 3.5" -or $_.DisplayName -EQ "Windows PowerShell ISE" -or $_.DisplayName -EQ "Windows Search Service" -or $_.DisplayName -EQ "Windows TIFF IFilter" -or $_.DisplayName -EQ "Wireless LAN Service" -or $_.DisplayName -EQ "XPS Viewer")}]]> https://docs.microsoft.com/archive/blogs/ken_brumfield/handy-sccm-updates-queries Thu, 09 Apr 2015 22:44:54 GMT https://blogs.technet.microsoft.com/ken_brumfield/2015/04/09/handy-sccm-updates-queries/ Context

Recently , while still basically focusing on platforms related technologies, I’ve been doing more and more work focusing on datacenter/cloud automation.  This obviously involves a lot more focus on integrating to accommodate business processes.  In short this is a collection of SCCM queries which I’m going to be maintaining as much for my own reference as anything else.

Note:  This will be updated as I come across new queries and or optimize the queries.

My thanks to all the many folks out there who had similar samples for me to build off of.

References

Systems needing updates which are included in baselines

Get a list of systems that need updates in the Software Update Groups, regardless of whether or not the update is actually deployed to the system.

Excluding unknown status

SELECT s.Name0, sn.StateName
    , CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN 402
        ELSE 500
    END As StateType
    , CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN uss.LastEnforcementMessageID
        ELSE uss.[Status]
    END As StateID
    , MAX(CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN uss.LastEnforcementMessageTime
        ELSE uss.LastStatusCheckTime
    END) As StateTime
FROM v_R_System s
INNER JOIN v_UpdateComplianceStatus uss ON s.ResourceID = uss.ResourceID
INNER JOIN v_CIRelation cr ON uss.CI_ID = cr.ToCIID
INNER JOIN v_AuthListInfo ali ON cr.FromCIID = ali.CI_ID
INNER JOIN v_StateNames sn ON
  CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN 402
        ELSE 500
    END = sn.TopicType
    AND
    CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN uss.LastEnforcementMessageID
        ELSE uss.[Status]
    END = sn.StateID
WHERE NOT ( NOT (uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0) AND (uss.[Status] = 1 OR uss.[Status] = 3))
GROUP BY s.Name0, sn.StateName
    , CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN 402
        ELSE 500
    END
    , CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN uss.LastEnforcementMessageID
        ELSE uss.[Status]
    END

Including unknown status

SELECT s.Name0, sn.StateName, MAX(uss.StateTime) As StateTime
FROM v_R_System s
INNER JOIN v_UpdateState_Combined uss ON s.ResourceID = uss.ResourceID
INNER JOIN v_CIRelation cr ON uss.CI_ID = cr.ToCIID
INNER JOIN v_AuthListInfo ali ON cr.FromCIID = ali.CI_ID
INNER JOIN v_StateNames sn ON uss.StateType = sn.TopicType AND uss.StateID = sn.StateID
WHERE NOT (uss.StateType = 500 AND (uss.StateID = 1 or uss.StateiD=3))
GROUP BY s.Name0, sn.StateName

Note:  The WHERE clause excludes ”Update Not Required” (StateID = 1) and “Update Installed” (StateID = 3)

Information About Updates Needed Per Computer (For which there are baselines)

This gives pretty much all the information needed about the patch state for all needed, unknown, error, and in process statuses.  This will also tell if the computer is in a collection that the update is deployed to.

Note:  At the database level, Software Update Group deployments get deployed on a per update basis (Assignments).  This means that by joining of v_CIAssignmentToCI and v_AuthListInfo to the CI_ID of the update will duplicate the rows.

All Needed Updates Excluding Unknown Status:  Simplified Version

SELECT s.Name0, ali.Title As [SoftwareUpdateGroup], ui.ArticleID, ui.BulletinID, ui.Title As UpdateTitle, ui.InfoURL, sn.StateName
    , MAX(CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN uss.LastEnforcementMessageTime
        ELSE uss.LastStatusCheckTime
    END) As StateTime, CAST(MAX(CASE WHEN tm.ResourceID IS NOT NULL THEN 1 ELSE 0 END) AS BIT) AS IsDeployedToThisComputer
FROM v_R_System s
INNER JOIN v_UpdateComplianceStatus uss ON s.ResourceID = uss.ResourceID
INNER JOIN v_CIRelation cr ON uss.CI_ID = cr.ToCIID
INNER JOIN v_AuthListInfo ali ON cr.FromCIID = ali.CI_ID
INNER JOIN v_StateNames sn ON
    CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN 402
        ELSE 500
    END = sn.TopicType
    AND
    CASE
        WHEN uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0 THEN uss.LastEnforcementMessageID
        ELSE uss.[Status]
    END = sn.StateID
INNER JOIN v_CIAssignmentToCI atci ON uss.CI_ID = atci.CI_ID
INNER JOIN v_CIAssignment a ON atci.AssignmentID = a.AssignmentID
LEFT JOIN v_CITargetedMachines tm ON uss.CI_ID = tm.CI_ID AND s.ResourceID = tm.ResourceID
INNER JOIN v_UpdateInfo ui ON uss.CI_ID = ui.CI_ID
WHERE NOT ( NOT (uss.[Status] = 2 AND uss.LastEnforcementMessageID > 0) AND (uss.[Status] = 1 OR uss.[Status] = 3))
GROUP BY s.Name0, ali.Title, ui.BulletinID, ui.ArticleID, ui.Title, sn.StateName, ui.InfoURL

All Needed Updates Excluding Unknown Status:  Performance Optimized Version (about 8x faster)

SELECT uss.Name0, aliloc.[DisplayName] As SoftwareUpdateGroup, uss.BulletinID, uss.ArticleID, loc.[DisplayName] AS [UpdateTitle], loc.CIInformativeURL As InfoURL, sn.StateName, uss.LastErrorCode
    , MAX(uss.StateTime) As StateTime, CAST(MAX(CASE WHEN cm.MachineID IS NOT NULL THEN 1 ELSE 0 END) AS BIT) AS IsDeployedToThisComputer
FROM
(
    SELECT uci.CI_ID, s.ResourceID, uci.ArticleID, uci.BulletinID, s.Name0
        , CASE
            WHEN cs.[Status] = 2 AND LastEnforcementMessageID > 0 THEN 402
            WHEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end) != 2 OR LastEnforcementMessageID IS NULL or LastEnforcementMessageID = 0 THEN 500
        END As StateType
        , CASE
            WHEN cs.[Status] = 2 AND LastEnforcementMessageID > 0 THEN LastEnforcementMessageID
            WHEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end) != 2 OR LastEnforcementMessageID IS NULL or LastEnforcementMessageID = 0 THEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end)
        END As StateId
        , CASE
            WHEN cs.[Status] = 2 AND LastEnforcementMessageID > 0 THEN LastEnforcementMessageTime
            WHEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end) != 2 OR LastEnforcementMessageID IS NULL or LastEnforcementMessageID = 0 THEN ISNULL(cs.LastStatusCheckTime, ss.ScanTime)
        END As StateTime
        , cs.LastErrorCode
    FROM CI_UpdateCIs uci
    INNER JOIN CI_ConfigurationItems (NOLOCK) AS ci ON uci.CI_ID = ci.CI_ID AND ci.CIType_ID IN (1, 8) AND ci.IsHidden = 0 AND ci.IsTombstoned = 0
    INNER JOIN Update_ComplianceStatus cs ON uci.CI_ID = cs.CI_ID AND cs.[Status] > 0
    LEFT JOIN v_R_System s ON cs.MachineID = s.ResourceID
    INNER JOIN Update_ScanStatus ss ON uci.UpdateSource_ID = ss.UpdateSource_ID AND s.ResourceID = ss.MachineID
) As uss
INNER JOIN v_CIRelation cr ON uss.CI_ID = cr.ToCIID
INNER JOIN CI_ConfigurationItems ali ON cr.FromCIID = ali.CI_ID AND ali.IsHidden=0 AND ali.CIType_ID=9
INNER JOIN v_LocalizedCIProperties_SiteLoc aliloc on ali.CI_ID = aliloc.CI_ID
INNER JOIN v_StateNames sn ON uss.StateType = sn.TopicType AND uss.StateID = sn.StateID
INNER JOIN CI_AssignmentTargetedCIs (NOLOCK) atci ON uss.CI_ID = atci.CI_ID
INNER JOIN CI_CIAssignments (NOLOCK) a ON atci.AssignmentID = a.AssignmentID
INNER JOIN Collections_G (NOLOCK) c ON a.TargetCollectionID = c.CollectionID
LEFT JOIN CollectionMembers (NOLOCK) cm ON c.SiteID = cm.SiteID AND cm.MachineID = uss.ResourceID
INNER JOIN v_LocalizedCIProperties_SiteLoc loc ON uss.CI_ID = loc.CI_ID
WHERE NOT (uss.StateType = 500 AND (uss.StateId = 1 OR uss.StateId = 3))
GROUP BY uss.Name0, aliloc.[DisplayName], uss.BulletinID, uss.ArticleID, loc.[DisplayName], loc.CIInformativeURL, sn.StateName, uss.LastErrorCode

All Updates Including Unknown Status:  Simplified Version

SELECT s.Name0, ali.Title As [SoftwareUpdateGroup], ui.ArticleID, ui.BulletinID, ui.Title As UpdateTitle, ui.InfoURL, sn.StateName
    , MAX(uss.StateTime) As StateTime, CAST(MAX(CASE WHEN tm.ResourceID IS NOT NULL THEN 1 ELSE 0 END) AS BIT) AS IsDeployedToThisComputer
FROM v_R_System s
INNER JOIN v_UpdateState_Combined uss ON s.ResourceID = uss.ResourceID AND NOT (uss.StateType = 500 AND (uss.StateID = 1 OR uss.StateID = 3))
INNER JOIN v_CIRelation cr ON uss.CI_ID = cr.ToCIID
INNER JOIN v_AuthListInfo ali ON cr.FromCIID = ali.CI_ID
INNER JOIN v_StateNames sn ON uss.StateType = sn.TopicType AND uss.StateID = sn.StateID
INNER JOIN v_CIAssignmentToCI atci ON uss.CI_ID = atci.CI_ID
INNER JOIN v_CIAssignment a ON atci.AssignmentID = a.AssignmentID
LEFT JOIN v_CITargetedMachines tm ON uss.CI_ID = tm.CI_ID AND s.ResourceID = tm.ResourceID
INNER JOIN v_UpdateInfo ui ON uss.CI_ID = ui.CI_ID
GROUP BY s.Name0, ali.Title, ui.BulletinID, ui.ArticleID, ui.Title, sn.StateName, ui.InfoURL

All Updates Including Unknown Status:  Performance Optimized Version (My testing shows a 45x improvement in query times):

SELECT s.Name0, aliloc.[DisplayName] As SoftwareUpdateGroup, uss.BulletinID, uss.ArticleID, loc.[DisplayName] AS [UpdateTitle], loc.CIInformativeURL As InfoURL, sn.StateName, uss.LastErrorCode
    , MAX(uss.StateTime) As StateTime, CAST(MAX(CASE WHEN cm.MachineID IS NOT NULL THEN 1 ELSE 0 END) AS BIT) AS IsDeployedToThisComputer
FROM v_R_System s
INNER JOIN
(
    SELECT uci.CI_ID, cm.ResourceID, uci.ArticleID, uci.BulletinID
        , CASE
            WHEN cs.[Status] = 2 AND LastEnforcementMessageID > 0 THEN 402
            WHEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end) != 2 OR LastEnforcementMessageID IS NULL or LastEnforcementMessageID = 0 THEN 500
        END As StateType
        , CASE
            WHEN cs.[Status] = 2 AND LastEnforcementMessageID > 0 THEN LastEnforcementMessageID
            WHEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end) != 2 OR LastEnforcementMessageID IS NULL or LastEnforcementMessageID = 0 THEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end)
        END As StateId
        , CASE
            WHEN cs.[Status] = 2 AND LastEnforcementMessageID > 0 THEN LastEnforcementMessageTime
            WHEN ISNULL(cs.Status, case when ss.ScanPackageVersion>=uci.MinSourceVersion then 1 else 0 end) != 2 OR LastEnforcementMessageID IS NULL or LastEnforcementMessageID = 0 THEN ISNULL(cs.LastStatusCheckTime, ss.ScanTime)
        END As StateTime
        , cs.LastErrorCode
    FROM CI_UpdateCIs (NOLOCK) uci
    INNER JOIN CI_ConfigurationItems (NOLOCK) AS ci ON uci.CI_ID = ci.CI_ID AND ci.CIType_ID IN (1, 8) AND ci.IsHidden = 0 AND ci.IsTombstoned = 0
    CROSS JOIN v_ClientMachines cm
    LEFT JOIN Update_ComplianceStatus (NOLOCK) cs on cs.CI_ID=uci.CI_ID and cs.MachineID=cm.ResourceID and cs.[Status] > 0
    LEFT JOIN Update_ScanStatus (NOLOCK) ss on ss.MachineID=cm.ResourceID and ss.UpdateSource_ID=uci.UpdateSource_ID
) uss ON s.ResourceID = uss.ResourceID AND NOT (uss.StateType = 500 AND (uss.StateId = 1 OR uss.StateId = 3))
INNER JOIN v_StateNames sn ON uss.StateType = sn.TopicType AND uss.StateID = sn.StateID
INNER JOIN v_CIRelation cr ON uss.CI_ID = cr.ToCIID
INNER JOIN CI_ConfigurationItems ali ON cr.FromCIID = ali.CI_ID AND ali.IsHidden=0 AND ali.CIType_ID=9
INNER JOIN v_LocalizedCIProperties_SiteLoc aliloc on ali.CI_ID = aliloc.CI_ID
INNER JOIN CI_AssignmentTargetedCIs (NOLOCK) atci ON uss.CI_ID = atci.CI_ID
INNER JOIN CI_CIAssignments (NOLOCK) a ON atci.AssignmentID = a.AssignmentID
INNER JOIN Collections_G (NOLOCK) c ON a.TargetCollectionID = c.CollectionID
LEFT JOIN CollectionMembers (NOLOCK) cm ON c.SiteID = cm.SiteID AND cm.MachineID = s.ResourceID
INNER JOIN v_LocalizedCIProperties_SiteLoc loc ON uss.CI_ID = loc.CI_ID
GROUP BY s.Name0, aliloc.[DisplayName], uss.BulletinID, uss.ArticleID, loc.[DisplayName], loc.CIInformativeURL, sn.StateName, uss.LastErrorCode

Determine Service Windows for Machines

Parse the binary field that stores the details of the schedule token and also what collection the service window comes from.

Reference:  http://myitforum.com/cs2/blogs/jhuston/archive/2007/07/30/sms-schedule-token-strings.aspx

SELECT
    s.Name0
    , c.CollectionName
    , CASE (sw.rawSchedules / POWER(CAST(2 AS BIGINT), 19)) & (POWER(CAST(2 AS BIGINT), 3) - 1)
        WHEN 1 THEN 'Effective only'
        WHEN 2 THEN 'Every '
            + CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 3)) & (POWER(CAST(2 AS BIGINT), 5) - 1) AS VARCHAR)
            + ' day' + CASE WHEN ((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 3)) & (POWER(CAST(2 AS BIGINT), 5) - 1)) > 1 THEN 's' ELSE '' END
        WHEN 3 THEN 'Every '
            + CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 13)) & (POWER(CAST(2 AS BIGINT), 3) - 1) AS VARCHAR) + ' week'
            + CASE WHEN ((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 13)) & (POWER(CAST(2 AS BIGINT), 3) - 1)) > 1 THEN 's' ELSE '' END + ' on '
            + CASE (sw.rawSchedules / POWER(CAST(2 AS BIGINT), 16)) & (POWER(CAST(2 AS BIGINT), 3) - 1)
                WHEN 1 THEN 'Sunday'
                WHEN 2 THEN 'Monday'
                WHEN 3 THEN 'Tuesday'
                WHEN 4 THEN 'Wednesday'
                WHEN 5 THEN 'Thursday'
                WHEN 6 THEN 'Friday'
                WHEN 7 THEN 'Saturday'
            END
        WHEN 4 THEN 'Every '
            + CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 12)) & (POWER(CAST(2 AS BIGINT), 3) - 1) AS VARCHAR)
            + ' month' + CASE WHEN ((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 12)) & (POWER(CAST(2 AS BIGINT), 3) - 1)) > 1 THEN 's' ELSE '' END
            + ' on the ' + CASE CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 8)) & (POWER(CAST(2 AS BIGINT), 3) - 1) - 1 AS VARCHAR)
                WHEN 1 THEN 'first'
                WHEN 2 THEN 'second'
                WHEN 3 THEN 'third'
                WHEN 4 THEN 'fourth'
            END + ' '
            + CASE CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 16)) & (POWER(CAST(2 AS BIGINT), 3) - 1) AS VARCHAR)
                WHEN 1 THEN 'Sunday'
                WHEN 2 THEN 'Monday'
                WHEN 3 THEN 'Tuesday'
                WHEN 4 THEN 'Wednesday'
                WHEN 5 THEN 'Thursday'
                WHEN 6 THEN 'Friday'
                WHEN 7 THEN 'Saturday'
            END
        WHEN 5 THEN 'Every '
            + CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 10)) & (POWER(CAST(2 AS BIGINT), 4) - 1) AS VARCHAR)
            + ' month' + CASE WHEN ((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 10)) & (POWER(CAST(2 AS BIGINT), 4) - 1)) > 1 THEN 's' ELSE '' END
            + CASE WHEN (sw.rawSchedules / POWER(CAST(2 AS BIGINT), 14)) & (POWER(CAST(2 AS BIGINT), 4) - 1)> 0
                THEN ' on day ' + CAST((sw.rawSchedules / POWER(CAST(2 AS BIGINT), 14)) & (POWER(CAST(2 AS BIGINT), 4) - 1) AS VARCHAR)
                ELSE ' on the last day of the month'
            END
    END + ' starting on ' + CONVERT(VARCHAR, sw.StartTime, 120) + CASE WHEN sw.UseGMTTimes = 1THEN ' (UTC) ' ELSE ' (System Local) ' END
    + ' for ' + CAST(sw.Duration AS VARCHAR) + ' minutes'
    As [Description]
FROM
(
    SELECT CollectionID, CAST(CONVERT(BINARY(8),'0x'+ Schedules, 1) AS BIGINT) As RawSchedules, StartTime, UseGMTTimes, Duration
    FROM CEP_ServiceWindows
) as sw
INNER JOIN [dbo].[Collections] c ON sw.CollectionID = c.CollectionID
INNER JOIN [dbo].[CollectionMembers] cm ON c.SiteID = cm.SiteID
INNER JOIN [dbo].[System_DATA] s ON cm.MachineID = s.MachineID

Get a comma delimited list of CI_IDs for each Bulletin and Article

Useful for automating approval of the updates and much faster than WMI queries.

SELECT
DISTINCT uci2.BulletinID, uci2.ArticleID,
STUFF
(
    (
        SELECT N','+  CAST(uci.CI_ID As NVARCHAR(MAX))
        FROM [dbo].[CI_UpdateCIs] uci
        INNER JOIN CI_ConfigurationItems (NOLOCK) AS ci ON uci.CI_ID = ci.CI_ID AND ci.CIType_ID IN (1, 8) AND ci.IsHidden = 0 AND ci.IsTombstoned = 0
        WHERE (uci.BulletinID = uci2.BulletinId AND uci.ArticleID = uci2.ArticleID)
        ORDER BY uci.CI_ID
        FOR XML PATH('')
    ), 1, 1, ''
) AS CI_IDs
FROM
(
    SELECT DISTINCT uci.BulletinID, uci.ArticleID FROM [dbo].[CI_UpdateCIs] uci
    INNER JOIN CI_ConfigurationItems (NOLOCK) AS ci ON uci.CI_ID = ci.CI_ID AND ci.CIType_ID IN (1, 8) AND ci.IsHidden = 0 AND ci.IsTombstoned = 0
) AS uci2

Using SCCM for patch deployment, all of sudden no status updates were getting posted by the clients.

Discovery

Reviewing WUAHandler.log showed the following error:

OnSearchComplete - Failed to end search job. Error = 0x8024000f.    WUAHandler    08/23/2014 1:53:10 PM    1104 (0x0450)
Scan failed with error = 0x8024000f.    WUAHandler    08/23/2014 1:53:10 PM    1104 (0x0450)

Now the question became why is the search job not ending?  Since SCCM uses WUAU to determine which updates are applicable, the next step was to see if there were any errors in the WindowsUpdate.log.  Reviewing the WindowsUpdate.log provided the following insight (trimmed for brevity):

2014-08-24    05:42:18:502     584    1a38    Agent      * WARNING: Exit code = 0x8024000F
2014-08-24    05:42:18:502     584    1a38    Agent    WARNING: WU client failed Searching for update with error 0x8024000f
2014-08-24    05:42:18:502     584    14e8    AU      # WARNING: Search callback failed, result = 0x8024000F
2014-08-24    05:42:18:502     584    14e8    AU      # WARNING: Failed to find updates with error code 8024000f

A quick review of Appendix G: Windows Update Agent Result Codes translated this to WU_E_CYCLE_DETECTED, "Circular update relationships were detected in the metadata."

Ahh, now I understand…  NOT!  Next step, as usual, is a quick internet search to see if someone figured this out for me.  Alas, no useful information came up, which is also why this makes it worthwhile to post (and so I'll be able to remember what I did the next time this happens).

Having spent some time in the WSUS database before I know there are a couple of tables building relationships between the various updates and/or update bundles.  For those who haven't spent a lot of time in here, the table that seems to be at the center of everything is [dbo].[tbRevision] which is linked to [dbo].[tbUpdate] via the [LocalUpdateID] column (did not design this, so just sharing what I've learned).  [RevisionId] from [dbo].[tbRevision] and [UpdateId] from [dbo].[tbUpdate] are used pervasively throughout the database and form the relationship between the abstract concept of the Update and the individual components that constitute said update.

What does "abstract concept of the update" mean?  For certain updates it there are multiple revisions of the  update in the database.  (again, don't ask me why, I didn't design this, just sharing what I've learned)  Note:  Digging deeper, both use the same files, so I suspect this may have something to do with metadata,  supersedence, expiration, etc.

Expanding on these two base tables via the Foreign Keys, and a little bit of intelligent discrimination (eliminating localized descriptions, etc. that I guessed a client wouldn't care about), I scoped this to the following potential loops:

• [tbRevision] –> [tbBundleAtLeastOne] –> [tbBundleAll] –> [tbRevision] – multiple revisions roll up into a parent revision (but don't necessarily need to).  Think ITIL concept of Configuration Item (CI), where a parent CI can contain multiple child CIs.  Brief review looks like this accommodates multiple binaries for scenarios where there are different OS languages that need to be patched.
• [tbRevision] –> [tbPrerequisite] –> [tbInstalledUpdateSufficientForPrerequsite] –> [tbUpdate] –> [tbRevision] – Self-explanatory, are the pre-requisites on the system?  I.e. for an update which covers multiple Service Pack, this would make sure the appropriate update is applied depending on which Service Pack revision is deployed.
• [tbRevision –> [tbRevisionSupersedesUpdate] –> [tbUpdate] –> [tbRevision] –  Save some time in by not installing the updates for which a later update covers the (security) fix and addresses the new issue.

Note that the above loops can theoretically also be combined, i.e. [tbRevision] –> [tbRevisionSupersedesUpdate] –> [tbUpdate] –> [tbRevsion] (different RevisionID –> [tbBundleAtLeastOne] –> [tbBundleAll] –> [tbRevsion] (pointing back to original RevisionID).

This allowed me to start putting together some SQL queries to figure out what might be going on.

SELECT *
FROM [dbo].[tbBundleAtLeastOne] balo
INNER JOIN [dbo].[tbBundleAll] ba ON balo.BundledID = ba.BundledID
WHERE balo.RevisionID = ba.RevisionID

No results…

SELECT *
FROM [dbo].[tbPrerequisite] p
INNER JOIN [dbo].[tbInstalledUpdateSufficientForPrerequisite] iusfp ON p.PrerequisiteID = iusfp.PrerequisiteID
INNER JOIN [dbo].[tbRevision] r ON iusfp.LocalUpdateID = r.LocalUpdateID
WHERE p.RevisionID = r.RevisionID

Still No…

SELECT u.UpdateID
FROM [dbo].[tbRevisionSupersedesUpdate] rsu
INNER JOIN [dbo].[tbUpdate] u ON rsu.SupersededUpdateID = u.UpdateID
INNER JOIN [dbo].[tbRevision] r ON u.LocalUpdateID = r.LocalUpdateID
WHERE rsu.RevisionID = r.RevisionID

Bingo!!!

A little more SQL assuages my curiosity of where this came from:

SELECT r.RevisionID, vu.UpdateId, r.RevisionNumber, CAST(CASE WHEN rsu.revisionID IS NULL THEN 0 ELSE 1 END AS BIT) AS IsSuperseded
    , lp.Title AS 'Company', vu.DefaultTitle, vu.DefaultDescription
, vu.ArrivalDate, vu.CreationDate, vu.IsDeclined, vu.PublicationState, vu.UpdateType
, vu.UpdateSource, vu.KnowledgebaseArticle, vu.SecurityBulletin
FROM [dbo].[tbUpdate] u
INNER JOIN [PUBLIC_VIEWS].[vUpdate] vu ON u.UpdateId = vu.UpdateID
INNER JOIN [dbo].[tbRevision] r ON u.LocalUpdateID = r.LocalUpdateID
INNER JOIN [dbo].[tbFlattenedRevisionInCategory] fric ON r.RevisionID = fric.RevisionID
INNER JOIN [dbo].[tbRevision] rCompany ON fric.CategoryID = rCompany.LocalUpdateID AND rCompany.IsLatestRevision = 1
INNER JOIN [dbo].[tbCategory] c ON fric.CategoryID = c.CategoryID AND c.CategoryType = 'Company'
INNER JOIN [dbo].[tbProperty] p ON rCompany.RevisionID = p.RevisionID
INNER JOIN [dbo].[tbLocalizedPropertyForRevision] lpr ON rCompany.RevisionID = lpr.RevisionID AND lpr.LanguageID = p.DefaultPropertiesLanguageID
INNER JOIN [dbo].[tbLocalizedProperty] lp ON lpr.LocalizedPropertyID = lp.LocalizedPropertyID
LEFT JOIN [dbo].[tbRevisionSupersedesUpdate] rsu ON r.RevisionId = rsu.RevisionID
WHERE vu.UpdateId IN
(
SELECT u.UpdateID
FROM [SUSDB].[dbo].[tbRevisionSupersedesUpdate] rsu
INNER JOIN dbo.tbUpdate u on rsu.SupersededUpdateID = u.UpdateID
INNER JOIN dbo.tbRevision r ON u.LocalUpdateID = r.LocalUpdateID
WHERE rsu.RevisionID = r.RevisionID
)

Resolution

Which now explains why this started right after I pushed some updates from SCUP. Note that SCUP was not the problem, it was the vendor who put the circular reference in the updated metadata.

1. Find the update in SCUP (search on the UpdateID).
    
   The two UpdateIDs I found:

• 06366964-77F9-4A48-9BAF-DBF9851BBAFF
• 9F4B59BC-E73C-4E02-A5E5-2B3B40A23D73

• Select the "Supersedes" Tab and confirm that it is Superseding itself.
  
• Select the update (under the "Superseded Updates" item) and select "Remove Update".
   
• Confirm that it is deleted.
   
• Re-Publish the update.
  This will “retire” the old revision and replace it with the new revision that is not superseded by itself.  Run the “big” query above to see how this happens.
  

Enter the PowerShell one liner:

&{$adapter = Get-NetAdapter -Name Ethernet;New-NetIPAddress -InterfaceAlias $adapter.Name -AddressFamily IPv4 -IPAddress 192.168.1.55 -PrefixLength 24 -DefaultGateway 192.168.1.1; Set-DnsClientServerAddress -InterfaceAlias $adapter.Name -ServerAddresses ("192.168.1.2","192.168.1.3")}

Using this, at least gets the server configured and dynamic DNS configured (if using it), so all you have to do is run an ipconfig /flushdns on your client and your remote PowerShell session should reconnect.

Key things to know:

• Requires PowerShell 3.0 – there are other examples out there for how to use PowerShell to invoke WMI to manage this for systems not yet on PowerShell 3.0.
• “;” indicates separate commands.  This is not piping data from one command to the other here, it is running 3 separate commands using the variable defined in the first.
• This isn't a script so there aren't any ExecutionPolicy considerations.
• Specify the adapter name, in this case “Ethernet” as appropriate for your system.
• Other commands to manage the TCP/IP settings are here:  Net TCP/IP Cmdlets in Windows PowerShell

Even though I exclusively work on the infrastructure side, I operate under the theory that if I do something more than once, I need to automate it.  As can be seen from the blog, I am spending more and more time exploring how to use System Center to minimize the amount of repetitive work I have to do.  Or at least do the same task in new ways to relieve the tedium.

Spending a lot of time in a lab, one of the challenges I have that scales to real world scenarios is that I constantly have to redeploy servers.  Since I have a wide variety of server configurations that I have to redeploy, I am continually looking for ways to reduce the amount of time I spend re-building.  For scenarios where I couldn’t find scripts that someone else was nice enough to share, my previous posts include links to some scripts I’ve built to help with these tasks.

Realizing that managing the library of scripts, trying to remember what I named the scripts in the past, manually  kick of the scripts, wasn’t the most efficient use of my time.  Meaning I needed to expand my knowledge of the tools available to for managing infrastructure.

My post “MOMCertImport – Is it all it’s cracked up to be?”, which included steps for configuring the servers via Application Management wasn’t quite as clean as I desired.  Specifically, I had to manage the script in multiple locations which meant updating multiple locations each and every time I updated the script.

Discovery

Goals:

• Validation that the system was properly configured prior to beginning any work efforts.
• Minimize the amount of time invested so that I could perform other tasks while the system was rebuilt.
• Increase the modularity of the script in order to minimize time and effort of future updates.
  Note:  I’ve seen some massive post build configuration scripts out there that are a nightmare to try and figure out what they do.  (my MOMCertImport script definitely falls into this category)
• Total build time is not a priority!  Total build involvement IS!

Research:

• PowerShell now has some really cool functionality for configuration management.  Unfortunately, this doesn’t come with a “platform” to deploy AND report on the configuration to the build servers.

• Here is a way to use DSC to automatically configure the system at boot:  http://blogs.msdn.com/b/powershell/archive/2014/02/28/want-to-automatically-configure-your-machines-using-dsc-at-initial-boot-up.aspx
  Note:  SCCM can also be used to deploy the configurations after the agent is installed.
• I did play around with this a little and it is really easy to work with.

• SCOM – I could have created a rule/monitor and response combination.  When it comes to managing groups of systems to scope rules, it is a little harder than SCCM.  Primarily that it is all bundled in MPs and the ability to delegate the group membership management has limited flexibility.
• SCCM Application Management – been there done that, see above.
• SCCM Compliance Management – simple, flexible interface that allows granular modularity of both compliance settings AND scope management.

Automation

There are 4 steps to doing this:

1. Create the script to detect and remediate the missing components.
2. Create the Configuration Item
3. Add the Configuration Item to a Configuration Baseline
4. Deploy the Configuration Baseline

1. The script

The script used is posted over on the TechNet Gallery:  Configure Exchange 2013 Server Role and Feature Dependencies.

As a basic explanation, there are 3 main sections:

• The list of Roles/Features required – This is the only portion that needs to be edited.  Just edit the list of features required here.
  Note:  If using the PowerShell Desired State Configuration use this reference:  Windows PowerShell Desired State Configuration Role Resource
• A comparison of the required list and how the machine is configured – shouldn’t need to edit
• Export of missing Roles/Features – shouldn’t need to edit

2.  Create the Configuration Item

• Setting type = “Script”
• Data type = “String”

Discovery Script

Make sure the remediate parameter is set to “False”

Remediation Script

Make sure the remediate parameter is set to “True”

Set the compliance rules

• Set “Rule type” to “value”.  Existential rules don’t allow for remediation.
• Set the operator to “Equals” and the value to (blank).  The current script design returns a null value if everything is installed.  This is my design, feel free to change it so the script outputs OK, 0, or something like that when the server is compliant.
• To automatically remediate “Check the box to “Run the specified remediation script when this setting is noncompliant.”  If it isn’t apparent, checking this box runs the script configured immediately preceeding this.

3. Add it to the Configuration Baseline

Note the revision column.  It’s kind of handy for managing the Configuration Items.  There can be one “Production” baseline that deploys a SPECIFIC version of the script and a “Development” baseline that deploys the latest/testing version.  That way trying to manage multiple, identically purposed Configuration Items can be avoided.
Why is this handy?  Have you ever gone into your script directory and tried to figure out which version of script.ps1, script.ps1.old, script-quicktest.ps1, etc. was the most recent tested version of what you were working with?  Now you have version control AND that means just by fiddling with the Revision the ability to roll back is provided.

4.  Deploy the Configuration Baseline

• Observe the setting “Remediate noncompliant rules when supported”.  To actually remediate, remediation needs to be enabled under the “Compliance Settings” AND the deployment configuration.  Meaning, only one Configuration Item needs to be created and actual remediation can be controlled.
• Remediation respects maintenance windows.
  Note:  Since all my servers “in production” should already be configured properly, I want this configuration baseline to remediate the servers as soon as possible after (re)installation.
• Schedule – there is a global client setting that controls how often Compliance Management is run, that can be overridden on a per deployment basis.

Other flexibility:

In a broader sense, if there is a scenario where the detection/remediation has to occur after an application is installed, there is an option to define that the application must be installed BEFORE the Configuration Item will start complaining.

In terms of Exchange, think of scenarios where you might want to have a member of the DAG rejoin the DAG after a reinstall.

WSUS Install and Configuration Script

From the run dialog (or command prompt) just execute “powershell –ExecutionPolicy Bypass” and it will start a PowerShell session that allows for running scripts and keeps the lowered permissions isolated to just the current running process.

Core Reference:  Authentication and Data Encryption for Windows Computers in Operations Manager 2007
Link to the script:  http://gallery.technet.microsoft.com/MOMCertImport-c3e7093b

Note:  While the below does work, it isn’t the Windows Server and System Center Product Groups official solution so if you can’t get it to work or have problems, please use the steps in the above reference.  That said, at the end of the day all MOMCertImport seems to do is import one used registry value.

Even though I tend to focus on platforms/OS related stuff, I’ve long since come to the realization that I can’t exclude tools such as System Center suite to help maintain and run a datacenter with high availability.  As such, I am constantly looking for how to leverage System Center to accomplish my platforms needs.  With things tending to be a little less hectic around the holidays I was able to spend some time solving some of those problems.

Not surprisingly the lab machines are frequently rebuilt.  In addition, having become utterly dependent on System Center for making sure the lab is relatively stable on a day to day basis and configured according to best practices.  This includes the use of both System Center Configuration Manager and System Center Operations Manager to deploy patches, deploy the monitoring agents, some other basic utilities, as well as basic monitoring.  The challenge in this lab are that there are 2 forests that do not have a trust.  While deploying the System Center Configuration Manager Client across the two un-trusted forests is a relatively easy exercise, getting the SCOM agents deployed to systems in the “remote” forest and talking to the Management Server has been a very painful exercise.

As many are aware, MOMCertImport is the officially provided tool to do this and there is plenty of content out there on how to use it.  The beef was that this process seems to have some rough edges and was very cumbersome every time a lab machine was flattened and redeployed.  The following are the sources of frustration every time there was rebuild:

1. Both of the forests have Certificate Authorities and use auto-enrollment for all systems in the forest.  Why can’t it just use the certificate that the machine automatically picked up on domain join?
2. Why is in necessary to create a custom template that had the exact same OIDs as the “Computer Template” that auto-enrollment used?
3. Why is it necessary to create a certificate with the Private Key marked as exportable and ship it around to get it installed?  Not only a hassle, but also has potential security and manageability implications.
4. Why is certificate enrollment such a cumbersome process?
5. Why is it necessary to have to have two certificates on each machine thus confusing every other application (and me when troubleshooting) on the systems that didn’t know how to handle two certificates with identical OIDs, subjects, etc?
6. In all fairness, a couple of the above challenges can be bypassed using the “/SubjectName” switch, however on a machine with multiple certificates with the same subject name MOMCertImport doesn’t always select the certificate that supports all the requirements for the SCOM agent.
   Note:  This wasn’t tested rigorously, but it seems to select the certificate with the SubjectName that was imported into the local machine store first.

Discovery

With those questions in mind it was time to start digging to understand what was going on when MOMCertImport is run.  This is what I found:

First, in just trying to dig up the instructions (for the umpteenth time) as well as troubleshooting the automation for importing the code below, the following were useful references as I proceeded:

• Troubleshooting OpsMgr 2007 and OpsMgr 2012 certificate issues with PowerShell
• http://thoughtsonopsmgr.blogspot.com/2009/09/momcertimportexe-tool-error-application.html

Being cautious with information, I looked to validate what I found.  I started investigating where I always do when it comes to trying to figure out what is happening on a system, good ol’ Process Monitor.  After setting the filter on Process Monitor as follows, here is what MOMCertImport.exe did:

The results were as follows:

Note:  I looked to see if any files were changed (filter on Operation “CreateFile” and “WriteFile”) and there were not any changes outside of MOMCertImport re-importing the certificate into to the store.

With remained was that MomCertImport created/changed two registry values:

• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Machine Settings
  • ChannelCertificateHash (REG_SZ)
  • ChannelCertificateSerialNumber (REG_BINARY)

The above blog entries focused on the ChannelCertificateSerialNumber registry value.  However, since there was a minor divergence between what was found in those reference and what Process Monitor uncovered, and I don’t know why MOMCertImport sets 2 keys but will work with only one, I decided to just copy the behavior of MOMCertImport.

Analysis

Looking more closely at the registry values unveiled the following:

• ChannelCertificateHash – This is the “Thumbprint” of the certificate.
• ChannelCertificateSerialNumber – This is the “Serial Number” of the certificate.  As the references above illuminated, this is a REG_BINARY value and has the byte values stored in the reverse order they are stored in the certificate.

This also solved all the problems I outlined above:

1. The certificates the computer gained through auto-enrollment can be used for the SCOM Agent.
2. No custom template needs to be created.
3. There is no need to have the Private Key be exportable or manually copy it around the environment.
4. This can be simplified by using pre-existing certificate enrollment automation.  Reference “Configure Certificate Autoenrollment” to enable
5. This can all be done with one certificate.
6. The Serial Number is what stored in the registry allowing for very precise selection of the certificate.

Furthermore, I found that MOMCertImport does no validation to determine if the certificate being imported meets the requirements of the SCOM agent.  In the automation below, it heavily leverages the content in the Troubleshooting OpsMgr 2007 and OpsMgr 2012 certificate issues with PowerShell article to institute validation before attempting to import the certificate.

Note:  There are a plethora of copies of articles out there how to do create/validate the certificates.  A large variance in the values for KeyUsage were found.  While I was able to get this to work with a KeyUsage value of 0xA0, as per the reference PowerShell troubleshooting script, the PG’s content on TechNet states 0xF0:  How to Obtain a Certificate Using Windows Server 2008 Enterprise CA in Operations Manager 2007.  I don’t know if this is a supported scenario, thus if using the PG reference, the “Computer” certificate template will need to be updated.  Also, I could use a smaller “KeyLength” than specified in the PG article, thus it seems the agent doesn’t care about the KeyLength.

Automation

There were several things that were necessary to automate:

• Validating existing configurations via SCCM.  Is there a certificate configured, installed, and valid for the SCOM Agent.  This IS SCCM’s detection method for the Application to configure the certificate.
• Select a valid certificate from the local store and configure the SCOM Agent to use it.
• Validating one or more certificates.

The first step towards doing this was to ensure that the above 3 tasks could be repeated on a per machine basis consistently.  To that end a PowerShell script seemed to be the best approach.  As can be seen from the comments on some of the previous blog posts, this blog platform doesn’t handle code and code formatting very well, plus the script is over 750 lines.  As such, the resulting script is uploaded on Technet’s ScriptCenter.  Go here to download it:  MOMCertImport.

The second step in the automation is to consistently get all the appropriate lab machines to run the scripted tasks.  Enter SCCM.

Creating the Application Deployment Type

Configuring Programs Properties

• Installation Program - powershell -ExecutionPolicy Unrestricted -File .\MOMCertImport.ps1 –InstallBestFitFromLocalStore
• Uninstall Program - powershell -ExecutionPolicy Unrestricted -File .\MOMCertImport.ps1 –Remove

Configuring Detection Method

• Set the Script Type to PowerShell
• Paste in the text of script above

Configuring Dependencies

• Obviously, configuring the certificate for the SCOM agent does no good if the agent isn’t installed.  Configure the deployment of the SCOM agent as a dependency of this Deployment Type.

That’s it!  Now just deploy the Application.

Note:  Initially I tried signing the script.  However it appears that when importing the script into the Detection Method SCCM does some reformatting.  As can be seen from the screenshot the script length is 658 lines whereas the script uploaded to TechNet ScriptCenter is 757 lines.  This reformatting breaks the signature.

• The line counts are for the initial posting, I’m sure there will be input on the script and the line counts will change over time.
• This has been filed as a bug on Microsoft Connect.  Reference:  CSS: ConfigMgr 2012 SP1 | App Evaluation fails when using a Signed Powershell Script as Detection Method when PS Execution Policy is set to AllSigned
• Workaround is to set the policy for the client settings to Bypass for the PowerShell execution policy.

Updates:

• 1/18/2014 – Fixed a bug in the Enhanced Key Usage Extension check.  It was passing everything if $EnableDiagnostics switch was used.
• 1/18/2014 – Added the ability to request a certificate from Microsoft Enterprise and Standalone CAs (untested against non-Microsoft CAs, but give it a try and let me know how it goes).
• 1/18/2014 - Now the file is so large it won’t cut and paste into the SCCM script validation.  However, if the “Open” option is selected and the script is loaded from a file it works fine.  This can also be solved by deleting the help content, the constants at the start, and everything in the #region Certificate/Request Install, as well as the labels in the “Main” switch statement for CreateCertificateRequest and GetCertificateResponse.  Yes, there is now some broken code in Main that references the certificate request functios, but it works.
• 1/18/2014 – This script WILL work against the SCOM server, but since the SCOM agent shouldn’t/can’t be installed on the SCOM server, the dependencies will prevent the script from running and will set the deployment state to “Not Applicable”.

Severity Type Site code Date / Time System Component Message ID Description
Error Milestone 002 01/09/2013 7:40:04 PM CM01.contoso.com SMS_DISTRIBUTION_MANAGER 2306 The source directory "\\contoso.com\SoftwareUpdates\c34e2458-681f-4a8b-8941-a460c2de314a" for package "0020000D" does not exist. The operating system reported error 2: The system cannot find the file specified.     Solution: Make sure you have specified a valid package source directory on the Data Source tab in the Package Properties dialog box in the Configuration Manager Console. If you specify a local path, it must be a local path on the site server. If you specify a path on a network drive, verify that the path exists and that the path is typed correctly.

I read the above message and said to myself, "Now, where'd it go?  Wait a sec... What is "it"?"

The challenge is that SCCM was tracking that the update had been download, but for some reason it wasn't in the location it was supposed to be.  When going into the console, setting the search criteria for "All Software Updates" to "Required >= 1" and "Installed >= 1" showed everything was downloaded according to SCCM.  Thus it took a little bit more digging for me to get this sorted out.  Since answers were sparse in regards to how to troubleshooting this outside of "review the logs", I figured I would share my solution.

While I still don't know WHY SCCM thought the update had been downloaded and yet the files were missing from the package source, I could at least figure out how to find the updates in the console to re-download them.

In the SQL query below just paste the highlighted (match the colors) from the above error message into the highlighted spots below.

DECLARE @MissingSourceDirectory NVARCHAR(512)
DECLARE @PackageId NVARCHAR(8)
SET @MissingSourceDirectory = 'c34e2458-681f-4a8b-8941-a460c2de314a'
SET @PackageId = '0020000D'

SELECT CASE
        WHEN ci.BulletinID LIKE '' OR ci.BulletinID IS NULL THEN 'Non Security Update'
        ELSE ci.BulletinID
        END As BulletinID
    , ci.ArticleID
    , loc.DisplayName
    , loc.Description
    , ci.IsExpired
    , ci.DatePosted
    , ci.DateRevised
    , ci.Severity
    , ci.RevisionNumber
    , ci.CI_ID
FROM dbo.v_UpdateCIs AS ci
LEFT OUTER JOIN dbo.v_LocalizedCIProperties_SiteLoc AS loc ON loc.CI_ID = ci.CI_ID
WHERE ci.CI_ID IN
(
    SELECT [FromCI_ID]
    FROM [dbo].[CI_ConfigurationItemRelations] cir
    INNER JOIN [dbo].[CI_RelationTypes] rt ON cir.RelationType = rt.RelationType
    WHERE cir.ToCI_ID IN
    (
        SELECT CI_ID
        FROM [dbo].[CI_ContentPackages] cp
        INNER JOIN [dbo].[CI_ConfigurationItemContents] cic ON cp.Content_ID = cic.Content_ID
        WHERE cp.ContentSubFolder = @MissingSourceDirectory AND cp.PkgID = @PackageId
    )
)

Qualification (Added 1/13/2013): It is important to be aware that the table CI_ConfigurationItemRelations can have multiple levels of relationships and there are different types of relationships. The above query worked well enough for me, so I didn't investigate further. I would suggest this reference for more details the CI_ConfigurationItemRelations table: Steve Rachui - ConfigMgr 2012–Application Model–Internals–Part I

Note: As a tip, I had several items missing from my source. I noticed in this case they were all from this month (January 2013), so after the third item month, I just went and changed the query in the console to show all "Downloaded" updates with "Date Released Or Revised" within the last month and downloaded them all as a batch.

Update 1/13/2013:

I realized I had a completely unnecessary piece of information in the original query. I had reused another query I had for which I needed the original date the update was released, not the date the most current revision was released. Below is the old query in case someone still wants to know the date the update was originally released.

DECLARE @MissingSourceDirectory NVARCHAR(512)
DECLARE @PackageId NVARCHAR(8)
SET @MissingSourceDirectory = 'c34e2458-681f-4a8b-8941-a460c2de314a'
SET @PackageId = '0020000D'

SELECT CASE
        WHEN ci.BulletinID LIKE '' OR ci.BulletinID IS NULL THEN 'Non Security Update'
        ELSE ci.BulletinID
        END As BulletinID
    , ci.ArticleID
    , loc.DisplayName
    , loc.Description
    , ci.IsExpired
    , orig_postdate.DatePosted as origDatePosted
    , ci.DatePosted
    , ci.DateRevised
    , ci.Severity
    , ci.RevisionNumber
    , ci.CI_ID
FROM dbo.v_UpdateCIs AS ci
LEFT OUTER JOIN dbo.v_LocalizedCIProperties_SiteLoc AS loc ON loc.CI_ID = ci.CI_ID
LEFT OUTER JOIN
(
    SELECT BulletinId, MIN(articles.DatePosted) As DatePosted FROM
    (
        SELECT ArticleId, MIN(DatePosted) As DatePosted
        FROM .[dbo].[v_UpdateCIs]
        GROUP BY ArticleId
    ) as articles
    INNER JOIN [dbo].[v_UpdateCIs] ci ON articles.ArticleID = ci.ArticleId
    GROUP BY BulletinId
) As orig_postdate ON ci.BulletinId = orig_postDate.BulletinId
WHERE ci.CI_ID IN
(
    SELECT [FromCI_ID]
    FROM [dbo].[CI_ConfigurationItemRelations] cir
    INNER JOIN [dbo].[CI_RelationTypes] rt ON cir.RelationType = rt.RelationType
    WHERE cir.ToCI_ID IN
    (
        SELECT CI_ID
        FROM [dbo].[CI_ContentPackages] cp
        INNER JOIN [dbo].[CI_ConfigurationItemContents] cic ON cp.Content_ID = cic.Content_ID
        WHERE cp.ContentSubFolder = @MissingSourceDirectory AND cp.PkgID = @PackageId
    )
)

What I choose to post here is information where I had a lot of difficulty getting the answers too.  I.e., it isn't floating around out there at all or doesn't answer some of the specific questions I needed answers too.  The challenge is that the format the answer will often work for me in is not necessarily something I can just cut and paste here.  So it has to wait until I have time to polish it, as I don't have Ralph Macchio hanging around to help (wax on, wax off).

Moving on to the topic of the post, if you didn't catch the double entendre in the title and the first paragraph, this post is about storage performance and design options.  This is from a scenario where there was fault tolerance was on each individual LUN and for capacity management they storage was being allocated in multiple small LUNs and concatenated at the server.  This was as a result of production outages due to poorly performing storage.  In working with the storage and SQL teams, this was a test that was run in response to the argument that “we don’t see performance improvements in striping over spanning”.  Which was absolutely true as in the test environment they were measuring only response times as a measurement of “performance”, not total throughput needed.  All us storage aficionados know that it is throughput (IOPS) demanded vs. the ability of the storage to deliver it which drives response times.

As an analogy, think of trying to get the high school football team to a game.  Let’s say it takes an hour to drive to the game.  Whether 1 mom/dad/coach takes a car with 3 of the players or one bus is taken with all the players, it still takes an hour to drive to the game.  This means multiple trips be made or multiple drivers have to drive.  Saying that the “trip” isn’t any faster doesn’t negate selection of the bus as the best option.

In short the conclusions below are really a reiteration of what we already know, more spindles exercised equals more throughput.  Spanning was essentially throttling full performance throughput of the storage to just the LUN which the active data was on.

For the critics out there, I know this isn’t real world and read/write ratios and higher costs of writes as well as aggregation of writes at the array controller impact total throughput.  The goal here is to explore the relationship between load driven, response times, where throughput maximizes while minimizing complexity of the test harness.  The relationship is what is important and will be consistent even if the storage configurations change.  (Note:  This is in the comments at the end, I put it here for all those who won’t read all the way to the end before posting feedback).

Striping vs. Spanning

Winner

Striping!!!

Return Of The Analysis

Overview

In a spanned set, data is only read to or written from the sub set of disks which hold the data needed. If all data is consumed all the time, this will eventually balance Input/Output (IO) as the storage fills. In the meantime, and for scenarios where only a subset of data is (think most recent month of 5 years of historical data in a database) only the spindles containing that data will be used.

For reference, minimal load to a fully loaded, but not overloaded disk, should respond to the operating system in 4 to 6 milliseconds (ms) on average, depending on the disk speed. Disk speeds will not go below 4 to 6 milliseconds due to physical limitations of the mechanical device. Therefore, as the IO requests from the Operating System and Application arrive at a rate greater than the storage can service the requests, said IO requests begin to wait in the queue. Thus the more requests that can not be serviced immediately, the greater the wait times. Degraded is considered to be in the 15 ms range, Critical in the 20 ms range.
NOTE: Cache will lower disk times, but caches WILL become saturated under sustained load in excess of what the storage can support and as such should not be included in planning for overall support load. Instead they should be looked at as an accelerator under normal load conditions and a buffer for transient load conditions. These tests were done WITH a cache on the SAN, so even if the belief that caches magically fix all evils, it can be observed here that there are still limits even on SAN and cache.

Note: ALL data below was configured on the same server on the same 3 LUNs, only the partition type was changed.

Legend

• Red Line is the disk latency discussed previously. (Avg sec/Read, Avg sec/Transfer, Avg sec/Write from PhysicalDisk or LogicalDisk performance counters)
• Blue Line is the number of operations per second performed by the storage (Reads/sec, Transfers/sec, Writes/sec from PhysicalDisk or LogicalDisk performance counters)
• Green Line is the number of operations outstanding. This is the independent variable and is controlled by the test harness (iometer). ("Current Disk Queue Length" from PhysicalDisk or LogicalDisk performance counters)

Spanning

Below (Figure 1) is the overall performance picture of the performance of the spanned system. As it is quite small, as specific areas are called out there will be a zoomed version near said text.

Figure 1

Observation #1

• Red line (latency) and green line (load generated) increase at the same rate.
• This supports the previous statement that disk wait times are correlated with the amount of IO being asked of the underlying storage.
• Real world application and idiosyncrasies:
• This is why "Current Disk Queue Length" is suggested as a counter to gauge whether or not storage is performing well. "Current Disk Queue Length" fails in that it is a point in time counter and can not accurately represent the median load over a given time period. Thus, when processing hundreds of IO per second (IOPS), one sample every second or greater it doesn't give a very good picture of the aggregate trend.
  NOTE: This problem scenario can be observed in the drops in the green line. Even under structured loads this data is skewed.
• One idiosyncrasy is that certain scenarios can cause the IO to be delayed "in flight" (somewhere between it exiting the queue and returning from the underlying storage). Low "Current Disk Queue Lengths" and high latencies can hint at this scenario. Due to the inaccuracies in "Current Disk Queue Length", the correct tools to confirm this scenario are native ETW tracing and tools, such as XPerf, that consume said data.

Observation #2

• Blue Line (IOPS serviced) peaks and stays flat regardless of how much more load the test harness attempts to push.
• Real world application:
  • Once you're done, you're done.
  • Can't squeeze blood from a stone.

Observation #3

• The Red Line (latency) in picture to the right (Figure 2) is scaled differently (1000x) so the actual values can be seen more clearly.
• As the latencies approach 20 ms, the maximum throughput approaches the absolute maximum.
• Reference the previous picture where the Blue Line (throughput) maxes out pretty close to the left hand side of the chart at about 800 IOPS.

Observation #4

• From the pictures (Figure 1 and Figure 3) below, performance maximizes at an average of about 800 IOPS

Figure 3

Striping

This is the same 3 LUNS reconfigured as a stripe.

Figure 4

Observation #1

• Red Line (latency) increases at a rate roughly equivalent to one-third the rate of increase of the Green Line (load).
• Again, this supports the previous statement that disk wait times are correlated with the amount of IO being asked of the underlying storage. The fact that the correlation isn't one to one is due to the fact that the OS sees 3 "physical disks" (each LUN is presented as a physical disk from the perspective of the OS) under this logical disk and the load is distributed across said disks. Thus each "physical disk" only sees one-third of the load, in turn only suffering one-third of the degradation
• Real world application and idiosyncrasies:
  • In addition to previously mentioned…
  • The performance at the logical disk level can be very different then the OS "physical disk" level. By spreading load across multiple "physical disks" the logical disk gains the advantages of the best and minimizes the consequences of the worst.

Observation #2

• Blue Line (IOPS serviced) climbs more slowly, but still eventually plateaus regardless of how much more load the test harness attempts to push.
• Real world application:
• Still can't get blood from a stone.

Observation #3

• The storage has to be pushed much harder to saturate it. In the spanned scenario, saturation was reached at about 16 pending IOs outstanding. In the striped scenario, this maxed out at about 48 pending IOs outstanding.
  Notice this is a factor of 3 greater than the striped scenario. This should not be a surprise.
• There appear to be 2 levels of saturation. One from 10 ms to 20 ms latencies and one from 20 ms and up. However, the higher level is much more volatile and "fails" down to the lower level of saturation quite often. This artifact should not be factored into scaling decisions.
• Same as above, changed the scaling on this picture (Figure 5) so the latency value is easier to read.

Observation #4

• From the picture below (Figure 6) the throughput maxes at about 2500+ IOPS.
• This is a little more than 3x the spanned scenario and should not be a surprise.

Figure 6

Testing strategy

Tools

IOMeter – www.iometer.org

Perfmon – included in Windows OS

Disk Manager – included in Windows OS

Configuration

• All Microsoft best practices were followed for storage configuration.
  • Partitions were aligned
  • File system used 64K clusters (as per SQL storage best practices)
• 3 LUNs were configured in A) Span and B) Stripe
• IOMeter
  • Access configuration – 64 KB IO sizes, 100% Random Read IO
  • Test Setup
    • Run Time – 60 seconds
    • Cycling Options – "Cycle # Outstanding I/Os – run step outstanding I/Os on all disks at a time
    • # of Outstanding I/Os – Start – 8, End – 256, Step 8, Linear Stepping
  • Iobw.tst (test file) was intentionally created to mostly fill one LUN (~32 GB) worth of space. This was selected to allow focus of analysis to be on the scalability of using one "physical disk" vs. using multiple "physical disks"
• Perfmon – The above IOMeter will create a test run approximately 35 minutes in length. Thus a performance counter log was created to automatically stop after a similar period (so the test could run unattended).
  • Collect all PhysicalDisk and LogicalDisk counters.
  • Sample in 10 second intervals - thus there are multiple data points for each IOMeter step and the steps can be observed

Comments

This was done to demonstrate the change in performance between striping and spanning, as well as illustrate the impact on changing load. As such, the IO profiles were simplified to exclusively be random read IO in order to present the worst case scenario (as random read IO has a very low cache hit rate) and minimize variability in results due to a cache optimizing writes to/from the storage. Therefore, the maximum throughput demonstrated in this test does not reflect the impact of write IOs. As a result, total throughput numbers will not be accurate for a real world production scenario, however the relationship between striping and spanning will remain similar. In short, the behavior pattern is able to be generalized, while the raw throughput numbers are not.

Additionally, scoping the test file size to reside on only one "Physical Disk" is not applicable to all scenarios. However, there are many scenarios where, due to data locality, this can easily be highly representative of real-time access.