What is the autorenewal procedure for multiple certificates enrolled using the same certificate template?

Question

Hi,

At one of our customers, I think we might be running into a certificate autorenewal bug on our Windows server 2019 Webservers which are configured to use SSL certificates with autorenewal.
Automatic rebinding of the certificate is enabled in IIS (via Task Scheduler) and SNI is activated for each website we host.

Deployment Scenario

1. On a single webserver we created a few websites.
2. Each website was configured with SNI enabled.
3. Certificate Autoenrollment was configured and applied via Group policy on the Webserver.
4. In IIS the 'Certificate Rebind' feature was enabled
5. For each website an SSL certificate was enrolled using the same custom 'Webserver' certificate template. The template had a lifetime of 1 year and a renewal at 2 months prior to end
6. The enrollment of all SSL certificates occurred in the same timeframe (less than 30 minutes)
7. Each certificate was bound to its proper website in IIS.

Renewal Time
At the moment of renewal, the group policy client side engine kicked off the autorenewal of the certificates. It should renew all certificates that will expire in less than 2 months as per the custom 'Webserver' template.
However, only one of all the certificates remained. And that one was used for rebinding to all websites, which was incorrect.

Investigation
When looking into the eventlog "CertificateServicesClient-Lifecycle-System", we noticed the events that took place to replace the old certificates.
we noticed the following with the first certificate:

1. Event 1006 was logged: A new certificate has been installed.
   The Enroll action informed us of the thumbprint of the new certificate
2. Event 1001 was logged: A certificate has been replaced.
   The renew action informed us of the old thumbprint and the new thumbprint.
3. Event 1005 was logged: A certificate has been archived
   The log entry shows the thumbprint of the certificate which has been replaced.

The subsequent action is to renew the following certificate. However:

1. No event 1006 is logged, we also see no certificate pending request at the CA.
2. No event 1001 is logged.
3. Event 1005 is logged, and the certificate is archived.

This happens for each subsequent certificate, and at the end the rebinding occurs with the single replaced certificate.

Reproducing the error
We have created a test-webserver and lowered the lifetime on the template to one day, with a renewal of 2 hours prior to expiration.
We are able to continuously reproduce the behavior.
Even when a certificate is not renewed at the same time but more than 2 hours later, no event 1006 nor 1001 is logged. The certificate not replaced but archived, and due to the lack of event 1001 automatic rebinding does not occur.
It only occurs for certificates based on the same template. During the test we also used single certificates based upon another template and those were replaced correctly.

Assumption

What we assume, based upon the tests we performed, is that in the renewal algorithm a verification occurs based upon the template oid/name.
it looks like the algorithm verifies that, if a certificate still exists in the certificate store that has a valid lifetime and is based upon the same template, other certificates that are also based upon this template do not need renewal.

Can someone have a look into this issue? Thanks!

Answer 1

I think we might be running into a certificate autorenewal bug on our Windows server 2019

it is not a bug, it is by design. Autoenrollment never was designed to handle multiple certificates based on same template where autoenrollment is configured. Only first instance of certificate is automatically renewed. If there are additional certificates you need to renew -- you must do it manually, or use scripting. I would recommend to read certificate autoenrollment whitepaper I wrote 3 years ago: https://www.sysadmins.lv/blog-en/certificate-autoenrollment-in-windows-server-2016-summary.aspx. This document explains the autoenrollment behavior in details and confirms that your issue is by design.

Answer 2

Hello @Stan Morisse ,

Thank you for posting here.

Hope the information provided by Crypt32 is helpful to you.

Should you have any question or concern, please feel free to let us know.

Best Regards,
Daisy Zhou

============================================

If the Answer is helpful, please click "Accept Answer" and upvote it.

Answer 3

Hi Vadims ( @Vadims Podāns ),

I have read through all the documentation you wrote, including the comments and answers.
I am already familiarized with most of it (except for some details), since I implement/configure/use this on a daily basis at my customers.

To ensure I didn't miss something, I have again re-read the article https://www.sysadmins.lv/blog-en/certificate-autoenrollment-in-windows-server-2016-part-2.aspx, which has a more depth view on certificate autoenrollment processing rules.

My findings based upon your documentation
To summarize what I have understood from that article, as per your article under chapter "Automatic certificate renewal":

1. In the first step, autoenrollment enumerates all existing certificates that use certificate templates and checks its validity.
2. If existing certificate passes validation checks, autoenrollment examines whether certificate template is set up for autoenrollment.
3. autoenrollment checks passes the certificate to certificate chaining engine (CCE) to determine its validity.
4. If existing certificate’s validity meets renewal threshold, autoenrollment will submit renewal request to CA server.

All of the above applies to my case, except for one important thing (and I assume that that is the 'by design' you are refering to ): The certificates were initially manually enrolled, as they required input for the website names.

Our case of manually enrolled certificates

When I look further into the chapter "Renew manually enrolled certificates" I've read the following:

Manually enrolled certificate renewal if none of the following conditions are true:
2. Existing valid and non-expired certificate based on this certificate template is found.

I assume that the latter is the 'by design' part of the process you are referring to. Once the first certificate has been renewed, the condition applies and therefore other certificates based on the same template are archived.

Remaining question

I certainly understand that you call it 'by design' if at all this is the exact case you're addressing me too.
However, in my opinion, a design can contain flaws. Earlier you wrote:

"Autoenrollment never was designed to handle multiple certificates based on same template where autoenrollment is configured".

Although I now certainly understand this to be true, the original document you refer to was for XP and the article does not contain the specific condition you described. So I was wondering where you got this information, regarding the conditions, from?

My 2 cents about this condition
According to me this 'by design' issue is something which MS should look into, as certificate use (TLS...) and autoenrollment becomes increasingly popular for private websites in Enterprises. Webserver SSL certificates in these webfarms are being set up with loads of certificates which should be able to be autorenewed without any manual or scriptbased solution, since that is exactly the definition of the word 'Autorenewal'.

Last but not least

1. Vadims (@vps ), I want to already thank you for pointing me in the right direction of your documentation, so I have some more understanding of the design.
2. Towards Microsoft: Regarding the design, I certainly would be a happy customer if Microsoft was able to have a look at this specific condition and rework it, so these certificates with different identities are not archived/deleted but renewed.

Thanks!