KB5040948 security patch crashed SQL Server

Question

KB5040948 security patch corrupted master database.

Script level upgrade for database 'master' failed because upgrade step 'ssis_discovery' encountered error 35262, state 4, severity 10.

It succeeded on standalone servers, but crashed on servers with availability groups and windows failover cluster.

**Update - I was able to bring the instances up from cmd, and everything is up, running and syncing. It even displays the patch version 15.0.4382.1. However, when I restart the server, the instances will not come up on their own. The WFSC comes back up, roles and nodes are online and everything appears to be running without issue as well.

***Update - The first run of the KB5040948 install followed by a reboot didn't install all of the components. Before I could try to uninstall, I had to run the KB5040948 install a second time followed by another reboot. When the server came back up this second time, I was able to run the uninstaller. Uninstalling the patch fixed the issue.

***Update - Attempts to apply CU 27 (KB5037331) encountered the same issue.  It appears the issue may not be with KB5040948 security patch but down chain with either CU 26 (5035123) or CU 27 (KB5037331) as these are the patches that had not yet applied prior to the KB5040948 security patch. 

---

Answer 1

Hey Guys,

I had encountered the same problem with applying KB5040948 update to our SQL 2019 AG environment.

The OS and SQL patching for the secondary node is automated and ran successfully, no issues. But applying the patch to the primary node raised the following alert on startup which in turn caused the instance fail:

Script level upgrade for database 'master' failed because upgrade step 'ssis_discovery' encountered error 35262, state 4, severity 10.

This is a serious error condition which might interfere with regular operation and the database will be taken offline.

If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting.

Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.

After several attempts to recover the system db's in the primary instance and reviewing others experience via Google search (none of those solutions resolved my issues), I advised on the following:

Connected to primary node server and opened the SSCM.

1. Add -T902 in startup parameter.
2. Started the SQL Server service successfully.
3. Removed the SSISDB from AG on primary node instance.
4. Removed -T902  from startup parameter.
5. Restarted the SQL service.
6. Executed below command to add SSISDB back to AG.

ALTER DATABASE SSISDB SET HADR AVAILABILITY GROUP = MyAG;

1. Checked the AG dashboard and confirmed that the AG is healthy and in sync.
2. Optional, monitor the environment to ensure stability.

 

Reference:

=================

Trace Flag 902

 

Using trace flag 902 is a quick workaround to bring up your SQL Server service if it fails to be started due to post upgrade script failure after installing SQL Server patch.

 

Please note that it is not suggested to keep your SQL Server service in this state for long since your SQL Server is in an incomplete patch installation status and considered instable for your production environment. This is because part of your SQL DLL files, MSP files or related registry key record might have already been upgraded while a few other files or metadata are not upgraded. 

 

SSISDB with AlwaysOn

 

SQL Server service packs usually run in single-user mode, while an availability database must be a [multi-user ]database. Therefore, during installing a service pack, you may find all availability databases including SSISDB will be taken OFFLINE and thus fail to patch.

 

To address this, you can first remove SSISDB from the availability databases, then patch each node (Failover Cluster Instances have to be in the [same patch level ]), then add SSISDB back.

 

Refer to https://techcommunity.microsoft.com/t5/sql-server-integration-services/ssis-with-alwayson/ba-p/388091.

Answer 2

Hi，bukkybu

This appears to be a known version bug and is consistent with what you described. Reference document:

https://learn.microsoft.com/en-us/troubleshoot/sql/releases/sqlserver-2022/cumulativeupdate6#2442348

Please try downloading the cumulative update package to fix this error.

Best Regards,

Mikey Qiao

---

If you're satisfied with the answer, don't forget to "Accept it," as this will help others who have similar questions to yours.

Answer 3

I suspect the root cause is the script msdb110_upgrade.sql which they run at installation of all CUs, GDRs etc. This script performs a lot of things of which some are quite dubious. And there are points where the error handling is so-so.

The root cause may be that you have some setting or condition on your server which is odd or even abnormal. But that should not crash the upgrade in my opinion.

I would recommend that you open a support case.

Answer 4

https://learn.microsoft.com/en-us/troubleshoot/sql/releases/sqlserver-2019/cumulativeupdate27#notes-for-this-update
If you enabled Always On together with the SSISDB catalog, see the information about SSIS with Always On about how to apply an update in these environments.

https://techcommunity.microsoft.com/t5/sql-server-integration-services/ssis-with-alwayson/ba-p/388091
Patching with Service Packs

SQL Server service packs usually run in single-user mode, while an availability database must be a [multi-user ]database. Therefore, during installing a service pack, you may find all availability databases including SSISDB will be taken OFFLINE and thus fail to patch. To address this, you can first remove SSISDB from the availability databases, then patch each node (Failover Cluster Instances have to be in the [same patch level ]), then add SSISDB back.