When I’m out on site with customers doing risk assessments we always see the same risks being raised, mainly about the following topics:

• Servers, especially DCs not being patched. For example we see MS14-068 missing on domain controllers in many customer environments. This patch (among others) is critical and was released in late 2014 – nearly 1 year ago. Please check your environment for missing patches.

• Too many members in the highly privileged admins groups – this one always flags and is down to there being too many people in Enterprise Admins and Domain Admins groups permanently or service accounts being a member of these groups

• High privilege users have password never expires set

• Stale user and computer accounts – very easy for attackers to compromise these accounts and stay unnoticed in an environment

• No configuration baseline or standard build for servers, especially DCs, therefore allowing unneeded, unwanted and potentially malicious software to run

• Allowing internet access from servers, particularly DCs

• Security baselines and basic security configuration not being done, such as restricting logon types to service accounts, restricting RDP access, other User Rights Assignment not being configured and the advanced firewall not being configured

It may seem too hard to solve some of these issues, however locking down these areas will significantly improve your overall security. It’s still a cat and mouse game but you should be trying to make this as hard as possible for the attackers and you really do need to make the investments financially and in the resource to implement and maintain these things.

This post will have links to Microsoft’s current recommended practice when it comes to security, mainly focussed on the identity space. Again, this post may evolve over time so please use as a reference point and check back.

A Snippet of Security as it relates to identity:

Immutable laws of Security

The following two links provide the 10 immutable laws of security V1 and V2, some great reference material for your CIO:

https://technet.microsoft.com/library/cc722487.aspx

https://technet.microsoft.com/en-us/library/hh278941.aspx

Also, see this link for some general links around Windows Security:

https://technet.microsoft.com/en-us/library/windows-server-security.aspx

Active Directory

The bible when it comes to AD security is the “Best Practices for AD Security” whitepaper written by MSIT and Microsoft consultants. This document includes all of the best practices for proactively managing and securing your directory based on experience from several security CritSits, AD Security Assessments and breaches of Microsoft customers. You should be looking to align your environment as best you can with these recommendations:

http://www.microsoft.com/en-gb/download/details.aspx?id=38785

Another great document to reference is the Threats and Countermeasures guide:

https://technet.microsoft.com/en-us/library/hh125921(WS.10).aspx

Credential Theft

Credential theft became a big problem a few years ago when Pass the hash attacks became prevalent. Although PtH has been around for many years it is only in recent years that tooling has been feely available that makes these attacks trivial to undertake. Although the PtH “vulnerability” cannot be fixed with an update there are plenty of mitigations to put in place and these are documented in the excellent PtH whitepapers which are a must read for any security admin:

https://www.microsoft.com/en-gb/download/details.aspx?id=36036

It should be noted that by mitigating these risks isn’t a panacea for security as the credential thieves will just use other techniques, but it does help address the low hanging fruit. There I said it, low hanging fruit J. Let’s see what other buzz terms I can drop in…

It’s also worth checking out the new Windows 10 feature call Credential Guard which also helps mitigate credential theft, again not a panacea but certainly raises the bar. There’s another one J

https://technet.microsoft.com/en-us/library/mt483740(v=vs.85).aspx

Accounts Security

Managing the security of accounts such as service accounts and accounts is paramount to the environments security as these accounts are usually high value targets for attackers. When using these accounts, the principle of least privilege should be used. Some guidance does exist on this and can be found here for service accounts:

https://technet.microsoft.com/en-us/library/cc170953.aspx

And here for admin accounts:

https://technet.microsoft.com/en-us/library/cc162797.aspx

When looking at service accounts it is also worth seeing if you can invest into Group Managed Service Accounts, introduced with Server 2012:

https://technet.microsoft.com/en-GB/library/hh831782.aspx

Lastly, check out the Best Practices for Delegating Active Directory Administration guide:

http://www.microsoft.com/en-gb/download/details.aspx?id=21678

PAM/JIT

Privileged Account Management and Just in Time elevation is quite big at the minute and managing accounts with high levels of access in the environment (either admin or VIP accounts) is crucial to maintaining good security posture. Therefore, Microsoft have implemented a PAM feature into Windows Server vNext and JIT feature into MIM 2016. Check this post for info on this:

https://technet.microsoft.com/en-us/library/dn903243.aspx

Also, our very own PoSH Chap Ian Farr has a good post on JIT using PowerShell and some of the built in features on Windows Server 2012:

http://blogs.technet.com/b/poshchap/archive/2015/05/29/latest-hsgb-outings-just-in-time-jit-administration.aspx

Proactive Monitoring and Detection

One of the key aspects of security is to perform monitoring, but also have the correct alerts in place tuned for your environment so you can use them to detect any attack or compromise. This guide will help you create this:

https://technet.microsoft.com/en-us/library/cc163158.aspx

Defining a Security Baseline:

You should have a security baseline in your environment for DCs, member servers and workstations such that you are blocking some legacy security protocols, blocking certain logon types for certain users, configuring the Windows firewall and services such as AppLocker. The Security Compliance Manager tool can help you with this:

https://technet.microsoft.com/en-gb/solutionaccelerators/cc835245.aspx

PKI

As you know from reading my blog PKI is my passion and the security around PKI including designing key signing ceremonies and secure implementations is what I specialised in before joining Microsoft. I’ve touched on some of the security aspects of PKI in my non-repudiation and offline CA virtualisation posts here:

http://blogs.technet.com/b/paranoidhumanoid/archive/2015/10/04/non-repudiation.aspx

http://blogs.technet.com/b/paranoidhumanoid/archive/2015/09/25/should-i-virtualise-my-root-ca.aspx

Further to my postings there is also the Securing PKI whitepaper which is very good:

https://technet.microsoft.com/en-us/library/dn786443.aspx

There are plenty of other reference guides and links that you can use, however the above will give you a starter for ten and plenty of bed time reading!

So here it is, loads of Group Policy and AD goodness here for your reference:

Stale Groups and Group Memberships

Use these scripts to identify groups with common members and also where these groups are assigned permissions on file shares.

http://blogs.technet.com/b/ashleymcglone/archive/2014/03/14/using-powershell-to-find-stale-and-duplicate-active-directory-groups.aspx

http://blogs.technet.com/b/ashleymcglone/archive/2014/03/17/powershell-to-find-where-your-active-directory-groups-are-used-on-file-shares.aspx

AD ACL Scanner

This PowerShell script is written by Robin Granberg, a Swedish PFE, who is the king when it comes to all things AD permission related and custom AD delegation. He is also one of the architects of the Premier POP–AD Delegation offering where we can help you implement a role based access control model in your environment, if you’re interested and are a Premier customer speak to you TAM!

https://adaclscan.codeplex.com/

GPO Debug Logging

Enabling GPO Debug Logging:

http://blogs.technet.com/b/mempson/archive/2010/01/10/userenvlog-for-windows-vista-2008-win7.aspx

Parsing the GPSvc log:

http://blogs.technet.com/b/askds/archive/2015/04/17/a-treatise-on-group-policy-troubleshooting-now-with-gpsvc-log-analysis.aspx

Enabling CSE Debug Logging

https://technet.microsoft.com/en-us/library/cc759167(v=ws.10).aspx

http://blogs.technet.com/b/askds/archive/2008/07/18/enabling-group-policy-preferences-debug-logging-using-the-rsat.aspx

Use this tool to parse the event viewer for Group Policy related events:

http://blogs.technet.com/b/grouppolicy/archive/2007/02/08/gplogview.aspx

List of Client Side Extensions

Group Policy Client Side Extensions can be a bit of a mystery because they are not very well documented. There isn’t one list in Microsoft where we can refer to each GPO setting and the CSE that is responsible for configuring it at the client side. The reason for this is because each product group is responsible for providing the Group Policy team with the settings and the client code to implement them in the form of a CSE. Group Policy is just the vessel for distributing these settings, it is not necessarily responsible for each of the settings, hence we don’t have a definitive list. However, never fear as Mark Empson, also known as Captain GPO, has documented some of the CSEs for us and what their GUIDs are:

http://blogs.technet.com/b/mempson/archive/2010/12/01/group-policy-client-side-extension-list.aspx

Slow Boot Slow Logon – are GPOs to Blame?

This isn’t technically related to AD and GPOs, however a lot of the time Group Policy gets blamed for slow booting and login times for user’s machines which is why I’ve included it here. If you read the Ask PFE Platforms blog, you are probably familiar with the series they did on SBSL, if not then you should be reading it! They talk through using Xperf and then Windows Performance Analyser for assessing boot times in Windows, this is mandatory reading for any client admins out there:

http://blogs.technet.com/b/askpfeplat/archive/tags/sbsl/

Folder redirection and roaming user profiles can eat up the seconds during logon for users, especially if you have slow storage or latency on the network. Can you replace folder redirection and RUP in your environment? Maybe. If you are licensed for MDOP then check out UE-V, it may fit your requirements:

https://technet.microsoft.com/en-us/windows/hh943107.aspx

Start-up scripts are also another culprit of slow boot times, check out this post to see if you can replace your scripts:

http://blogs.technet.com/b/askpfeplat/archive/2015/04/27/the-startup-script-is-dead.aspx

If you can replace the old login script which performs drive maps using Group Policy Preferences, then check out this post which will help you configure it:

http://blogs.technet.com/b/askds/archive/2009/01/07/using-group-policy-preferences-to-map-drives-based-on-group-membership.aspx

Further to above posts see some general guidance on group policy and logon times:

http://blogs.technet.com/b/grouppolicy/archive/2013/05/23/group-policy-and-logon-impact.aspx

Consolidate Policies

The following PowerShell script can be used to merge policies where you are trying to consolidate:

http://blogs.technet.com/b/ashleymcglone/archive/2011/01/19/finally-copy-and-merge-gpos-powershell-saves-the-day.aspx

Central Store

Have you implemented the Group Policy Central Store? If not check out this article:

http://blogs.technet.com/b/askpfeplat/archive/2011/12/12/how-to-implement-the-central-store-for-group-policy-admin-templates-completely-hint-remove-those-adm-files.aspx

If you have implemented it, are you having issues due to inconsistencies with ADMX versions and deprecated settings? If so, check out this hotfix which allows you to override the Central Store on a particular machine:

https://support.microsoft.com/en-us/kb/2917033

Execute those legacy ADM files from your SYSVOL using this article and script:

http://blogs.technet.com/b/askpfeplat/archive/2012/03/14/central-store-and-adm-removal-q-amp-a-with-an-updated-script.aspx

Backup, Backup, Backup!!!

Do you backup your AD, probably yes. Do you have a full forest recovery and disaster recovery plan, probably not. So many customers neglect the need to have a fully tested and robust disaster recovery plan in place for AD and GPOs. AD usually underpins your whole environment; you MUST create a DR plan for it! Go through the Forest Recovery whitepaper on TechNet and create one if you haven’t got one already:

https://technet.microsoft.com/en-us/library/planning-active-directory-forest-recovery(v=ws.10).aspx

With regards to GPOs, these will be included in your DR plan when you have it but most customers don’t take backups of their GPOs properly. Our very own PoSH Chap Ian Farr can help us out here: http://blogs.technet.com/b/poshchap/archive/2014/04/25/comprehensive-gpo-backup-script.aspx

Again, if you are Premier customer we have an engagement which can help you create your DR plan, ask your TAM about an AD Recovery Execution Service (ADRES).

DNS

So many customer do not enable scavenging on their DNS zones, or have it miss-configured, please see this article for information on how to configure DNS scavenging: http://blogs.technet.com/b/networking/archive/2008/03/19/don-t-be-afraid-of-dns-scavenging-just-be-patient.aspx

Have you upgraded your environment from Windows 2000 to 2003 to 2008 onwards? If so, did you migrate your _msdcs zone into the application partition? If you didn’t then you should, this still gets picked up in some customer environments today. Please see this article for information on how to configure the _msdcs zone to be in the forestdnszones app partition: https://support.microsoft.com/en-us/kb/817470

Have you configured secure dynamic updates on your DNS zone? This is another one which comes up on the risk assessments quite a lot, see the following article for info on secure updates:

https://technet.microsoft.com/en-us/library/cc961412.aspx

AD Replication

Lingering objects are an issue in many environments, you need to remove these and fix your replication for a healthy directory:

http://blogs.technet.com/b/askds/archive/2014/09/15/remove-lingering-objects-that-cause-ad-replication-error-8606-and-friends.aspx

A nice tool to monitor AD replication:

http://blogs.technet.com/b/askds/archive/2012/08/23/ad-replication-status-tool-is-live.aspx

If you’re feeling conflicted, then check out another post from PoSH Chap:

http://blogs.technet.com/b/poshchap/archive/2015/03/27/one-liner-use-powershell-to-find-forest-conflict-objects.aspx

Time Configuration

Many customers do not consistently configure their Root PDCe to point to an external time source. Follow this article to do this automatically with Group Policy:

http://blogs.technet.com/b/nepapfe/archive/2013/03/01/it-s-simple-time-configuration-in-active-directory.aspx

Stop devastating time drifts in your directory by configuring the max negative and positive phase correction values, again this is missed by many customers:

http://blogs.technet.com/b/askpfeplat/archive/2012/11/23/fixing-when-your-domain-traveled-back-in-time-the-great-system-time-rollback-to-the-year-2000.aspx

Post 2003 Eradication

Have you finally got rid of that last 2003 domain controller in your environment? If so, then make sure you perform the post upgrade tasks and don’t just close the project down:

Raise the DFL and FFL:

http://blogs.technet.com/b/askds/archive/2011/06/14/what-is-the-impact-of-upgrading-the-domain-or-forest-functional-level.aspx

Migrate SYSVOL to DFSr

http://blogs.technet.com/b/filecab/archive/2014/06/25/streamlined-migration-of-frs-to-dfsr-sysvol.aspx

Make use of the AD Recycle Bin:

http://blogs.technet.com/b/canitpro/archive/2014/05/01/step-by-step-enabling-active-directory-recycle-bin-in-windows-server-2012.aspx

Enable the GPO Central Store as discussed above.

Sites and Subnets:

Pretty much every single risk assessment I run the issue of missing subnets gets flagged, use this script to collect missing subnets in AD:

https://gallery.technet.microsoft.com/scriptcenter/Collect-Active-Directory-fa4dff2e

Authentication

Having problems with Kerberos? It could be down to duplicate SPNs and UPNs, check this script to find those duplicates:

https://gallery.technet.microsoft.com/scriptcenter/Find-Specific-or-Duplicate-b9c04d01

Are you suffering from token bloat? If so removing sIDHistory data may help, use this script:

http://blogs.technet.com/b/ashleymcglone/archive/2011/11/23/how-to-remove-sid-history-with-powershell.aspx

For reference, the non-repudiation check box I’m referring to is this one, which you can get to from the extensions tab on a certificate template:

It seems easy enough to provide non-repudiation by simply ticking this box on a certificate template right? Well, no it isn’t, even though it appears on the surface that you have non-repudiation, it’s a little more complicated than that.

What does it all mean?

So what is non-repudiation? The pure definition is as follows:

Nonrepudiation is the assurance that someone cannot deny something. Typically, nonrepudiation refers to the ability to ensure that a party to a contract or a communication cannot deny the authenticity of their signature on a document or the sending of a message that they originated.

What does this mean in terms of PKI then? Essentially what we are saying when we check that box on the certificate template is that the owner of that certificate, when they sign some data, cannot refute the fact that they were the original signer. This is actually quite a big deal and puts a lot of onus on the user and certificate supplier to ensure that these terms can be met.

Why do we need it?

Non-repudiation is more of a legal concept than one which applies to PKI. Therefore, if we want our digital signatures to be able to be held up in a court of law non-repudiation plays a big part in this. Such as within law enforcement if an officer was going to digitally sign a statement. It’s also used for attribution; this can be very important in some circles where the owner of a system needs to be absolutely sure of the authenticity of a message or that we can attribute the origin of the message undoubtedly to the original signer. Ultimately it builds a level of trust with the certificate.

So how do we achieve it then?

Non-repudiation is all about process and procedures, there are some technical controls that need to be in place but mainly it’s down to policy. What we need to ensure is that there is no way that any other entity other than the intended subject of the certificate is able to get a certificate in the subject’s name. For example, if Joe Blogs has a certificate which asserts non-repudiation there should be no way that Jane Blogs should be able to enrol for a certificate that has the subject name of Joe (and is within the same trust hierarchy). If she can then she would be able to sign some data as Joe and therefore non-repudiation is broken. We must also make sure that no one other than Joe can use the private key associated with Joes certificate. For example, if James can somehow gain access to Joe’s key he can sign some data as Joe, again breaking non-repudiation.

The way we do this is all down to the controls we put around our certificate authority and the enrolment processes that we use for the subject. This needs to be absolutely air tight from the Root CA down, such that we can be sure beyond any reasonable doubt that the PKI hasn’t been compromised in any place. If the root is compromised then anything underneath it can be deemed to be compromised and therefore untrustworthy, same from the subordinate CA down. Once that chain of trust is broken so is non-repudiation and the value of the end entity certificates almost becomes worthless.

How do we implement a PKI whereby we can be assured beyond any reasonable doubt that there hasn’t been a compromise, including for the enrolment process of the end entities? The answer is “with great difficulty”, or least with a lot of investment in the security and procedural controls around a PKI which, unfortunately, most organisations don’t want to invest in unless they are a commercial CA or government agency. I won’t go in to excruciating detail on all of the required elements for this but I do want to lay out a framework which needs to be in place.

Certificate Policy and Certificate Practice Statements

First and foremost, and before we build any of the PKI we need to document the technical and procedural controls that we wish to implement in a CP and various CPS documents for our PKI. At a very high level the CP is the document which describes the policies we wish to define in our PKI and contains things such as the enrolment methods which must be used for particular certificate types and how the private keys of the CAs must be protected. For example, we may say that we are issuing high assurance certificates to end entities and their private key must be stored on a smartcard. There can be one or more CP documents in a PKI depending on what policies you wish to apply to your CAs.

The CPS document is associated directly with a CA and describes how a particular CA meets the policy requirements as outlined in the CP. For example, we may say that this CA uses a Hardware Security Module to satisfy the control within the CP which discusses CA private key storage. Generally speaking, each CA that is implemented should have its own CPS as each CA may be managed differently or conform to different policies or elements of a policy.

A policy management board should also exist to ensure internal oversight of the PKI and they should be informed at all stages of implementation and whenever operation of the offline CAs takes place as well as at some instances of operation at the online CAs.

CP’s and CPS’s are also commonly misunderstood within a PKI and how we can build a PKI and associate these policies. I have some thoughts on this I wish to post in the coming weeks as well.

Security Controls Around the CA’s

The security controls of the CA should be documented in the CPS and will include a lot of different elements, I want to touch very briefly on some of the controls you should think about. This is by no means an exhaustive list of all the controls but provides a minimum list of criteria:

Root CA:

• Should be offline and never connected to any network. This is to ensure that no network based attack can be undertaken or malicious actor can connect to the CA. With it being offline we can be assured that only people with physical access to the CA can log on and perform operations at it, therefore limiting its attack surface. This is why I think virtualised offline CAs are a bad idea.

• The CA should be protected with sufficient physical security controls. Typically, this means in a restricted datacentre, with nominal controls such as a protected building, high fences, security guard, CCTV, authentication to the server room (biometric or access card) and finally a secured rack or room where only authorised people have access.

• Only authorised people should have physical and logical access to the CA. These people should be nominated teams or individuals within the organisation who have some authority. For example, the head of IT and the information security manager. It is good practice to implement dual person control such that both these people or their representatives need to be present at the same time to access the CA both physically and logically. Think about this as they show in the movies when launching a nuclear missile, two people of authority need to be present with their keys and insert and turn them at the same time. This stops one person having ultimate control of the asset meaning more than one person needs to be coerced for a compromise. This obviously needs to be implemented such that there is redundancy in the access control, if only two people have access and one goes on holiday then you lose operational access to the CA. The organisation needs to define the roles and responsibilities for the CA access. In terms of logical access control, a good method is to have a long complex administrator password to the root CA and split this into two halves, say 12 characters each and then each representative has one half of this password. It would also be prudent to encrypt the local hard drive of the CA to stop offline attacks to the server.

• The CA’s private key should be protected and stored in a Hardware Security Module. There are some very important reasons for this. Firstly, the HSM device provides a very robust environment where the key is stored, in a tamper evident and in some instances tamper resistant piece of hardware. The key is only ever in plain text within the confines of the HSM and cannot be extracted and ex-filtrated from the box. All signing operations are done within the secure hardware. The HSM provides a hardware based random number generator that provides excellent entropy for key generation and also accelerates signing operations because this is all done using a dedicated crypto-processor. Lastly, the HSM provides excellent access control mechanisms for key operations, such as requiring k of n people to load and operate the private key. This means that out of a total of n people k people are required to perform the operation, so we can set this to say 3 of 5 people. Again this limits collusion between people and means many people will need to be coerced to compromise the integrity of the CA.

• The CA should be implemented following scripted and robustly documented procedures in the format of a key signing ceremony. Unfortunately, this does not include party poppers and hats but is instead a very strictly controlled procedure which is usually audited by a third party. The third party audit gives oversight to the implementation and ensures the organisation follows the exact procedure they have defined in the documentation and this isn’t deviated from. Some of the roles within a key signing ceremony are as follows:

  • Auditor / Key Ceremony Observer – responsible for third party oversight

  • Key Ceremony Directory – responsible for the overall security of the ceremony

  • Policy Management Board Representative – this person is responsible to the policy management board and reports the progress and status of the PKI implementation or operation

  • Key Component Holder – there will be many key component holders and they are responsible for the secure retention and usage of the key component during the ceremony. For example, this may be a HSM card or passphrase that is used. The k of n principle is enforced by these people

  • Key Component Courier – these people are responsible for the secure transportation of the key components before and after the ceremony. The components should be stored in tamper evident containers, for example police evidence bags.

  • Key Component Custodian – these people are responsible for the secure storage of the various key components and the signing in and out of the component to the courier.

  • CA Operator – this person is the technical operator of the CA and performs the actual role installation and technical tasks according to the documentation.

If you’re really interested in key singing ceremonies check out this video https://www.youtube.com/watch?v=b9j-sfP9GUU which was recorded to demonstrate the controls and procedures that were put in place for the signing of the Root DNS zones for DNSSEC. The concepts in the video are entirely applicable to a PKI key signing ceremony.

As you can see the process and procedures of the installation of the CA far outweigh the technical elements of performing the role installation in server manager. The above only touch on the main points of the CA security, it can be far more intricate than this and requires a lot of documentation and patience to do it right!

The online CAs will have similar controls, such as the HSM, physical access controls and implementation ceremony. However, as the CA is fully online, i.e. network connected, it will require extra security controls around its logical security posture. A hardened build should be used, with the firewall enabled and network security controls in place. Also, the administration of the CA should be limited to only the required administrators and a robust Privileged Identity Management practice should be in place (check out MIM 20126 and Windows Server 2016 for this). The CA is a tier 0 system and therefore good credential hygiene should be in place and security management practice for this and the underlying AD should be conforming to the current recommended practice. See the following whitepapers for help on this:

• Best Practices for Securing AD https://technet.microsoft.com/en-us/library/cc773365(v=ws.10).aspx

• Best Practices for Delegating AD Administration http://www.microsoft.com/en-us/download/details.aspx?id=21678

• Privileged Identity Management with MIM: https://technet.microsoft.com/en-us/library/mt150258.aspx

• Mitigating Pass the Hash: https://www.microsoft.com/en-gb/download/details.aspx?id=36036

By managing your PKI and other core infrastructure systems to these high standards we can be sure, beyond reasonable doubt, that a compromise of the infrastructure is very, very, unlikely. It isn’t impossible but the bar is raised significantly.

Enrolment Strategy

Once you have a well-managed, securely implemented and operated PKI you need to be able to get some certificates from it for your end entities. This could be for things like encryption services, authentication and crucially digital signature with non-repudiation. What good is a PKI if it can’t issue certs? The actual enrolment process for end entity certificates needs to be as robust as the PKI controls we have spoken about. Auto-enrolment does not cut it for non-repudiation, we need to have a verification process for our subjects that usually involves the four eyes principle, i.e. two people, one to create the request and one to approve it. How can we place any value in the end entity certificates if the only verification for the subject is a letter from their mum? Let’s look at some of the controls we need to put in place:

• The subject’s identity must be verified using some form of physical ID. In the UK we have a standard for identity checks to government systems called e-Gif Level 3. This normally requires the presentation of two forms of ID, nominally a driving license and passport. Therefore, when the subject enrols for their certificate the identity information in it is as good as the ID presented, in this case two forms of government issued ID.

• Sponsorship is another must. We can’t let anyone enrol for a certificate from the PKI. We need to ensure that the subject is authorised to enrol, we can do this by ensuring they have sponsorship from their manager, business owner or other person in authority.

• I mentioned the four eyes approach. This provides split responsibility for the enrolment of the certificate, again ensuring collusion of two or more people is necessary to subvert the security controls of the CA. This can be done by having one person or role which is responsible for the ID verification of the user and the generation of the request and another role or user who is responsible for the actual issuance of the certificate. The enrolee should be present during the request creation and during the issuance to ensure that only the subject of the certificate receives the certificate from the CA.

• Only the subject of the certificate should ever be in control of the private key associated with the certificate. This is ultimately achieved using a smartcard. A smartcard is absolutely necessary for non-repudiation and they keys must be generated on card. The principle of a smartcard is identical to the principle of a HSM. The smartcard has its own crypto-processor and storage area for the key material and is deemed to be secure as the key cannot be extracted from the card (yes there are some side channel attacks on smartcards but they are the most secure key storage we have for users). If the key is in software stored on a Windows machine, then any administrator of that machine can export that private key and then you break non-repudiation and the trust of that certificate. It is also essential that the private key is generated on the card and never backed up. We need to be certain that only the owner of the smartcard is ever in control of the private key and that it is never present anywhere else or observed by any other person or system.

Summary

Can we say we have non-repudiation by putting a check in a box on a certificate template? Absolutely not, we must first jump through many hoops to be sure that only the owner of a private key associated with the certificate ever has access to it. This involves many controls, policies, procedures and security practices, some of which are listed above. Is this article the definitive list for everything you must do to secure a PKI, absolutely not, but hopefully it gives you some insight into what you need to do to implement a robust PKI. If implemented correctly we get non-repudiation but also the trust and assurance we place in all certificates has a higher value and we can have very strong two-factor authentication to internal systems.

A good friend of mine Andy Lyon has a brilliant quote I’m going to steal as I think it is apt for this post:

“CAs are easy, PKI is hard”

TLDR

Yes, we can set up a secured virtual environment and configure elaborate solutions for our CA to meet the required audit compliance or security requirements, but why bother? Just because you can do something does not mean you should. So my advice is DON’T DO IT!

The rest of this post will explain my reasoning’s for this.

The Proposal

In many environments a PKI underpins the security of the network, the user identities and access to systems and data. It can be the most trusted system an organisation owns, and if deployed in the correct manner will provide extremely robust authentication for devices, systems and users. It can also provide digital signature capability for signing code and signing emails or other documents to provide assurance around the integrity of said code or documents. If implemented correctly a PKI can also provide non-repudiation for signatures. Non-repudiation is where the signer of a document or other piece of data cannot refute that they were the original signer, i.e. I couldn’t send an email, sign it with my private key and then state that the email originated from someone else or that they had impersonated me. This is crucial for obtaining correct attribution for actions, but also signatures used in this way can, under some circumstances, be held up in a court of law. I’ll have another post in the coming weeks about non-repudiation, it takes much more process and policy than just a check in a box on a certificate template and therefore I think it requires its own post.

Don’t get me wrong, PKI can be implemented in an enterprise environment to deploy certificates where we place a low assurance value in those certificates. This is absolutely fine where the organisation knows this and isn’t relying on those certificates to provide authentication to High Value Assets, secure critical networks or provide robust signatures as stated above. You need to understand the purpose of your PKI and why you deployed it. For example, it’s no good implementing an accessible fully online PKI with no process or policy and then issuing smartcards for authentication from this PKI and thinking they provide highly robust authentication for your users, this is absolutely not the case and if not done correctly can even provide a lower level of security than a random long password (I also have some thoughts pending on smartcards to write up soon as well).

For more information about securing a PKI please see the Microsoft whitepaper at https://aka.ms/securingpki.

So, we have now set the scene for why PKI is important. Now down to design and implementation and the crux of why I put this post together. Many administrators when building a PKI design a simple two-tier deployment (my recommendation for 99% of customers) as per the following diagram:

The best practice is to have the Root CA offline and not connected to any networks to protect the sanctity of this server. This is a well understood practice and adhered to by many PKI administrators. It is also recommended to use a hardware device for protection of the private key material on the CA, i.e. to use a Hardware Security Module or HSM. In the typical scenario above the offline server would be disconnected from the network and usually stored on a physical server or other hardware such as a desktop PC or laptop. It would also have a HSM connected to it, either via a PCIe card or today we can use a USB connected HSM. The reasoning for this is all down to security and lowering the surface area for attack while also providing access control and robust auditing measures. The issuing servers are usually fully online and can be hosted on a virtual platform (where that platform is secured and managed to the same security tier as the CAs – usually tier 0) and are generally connected to a network based HSM.

So at this point you may be asking yourself “what is this guy’s problem then?”. Ok, so my problem is that we have this very well understood implementation methodology which is based loosely on the design principles above which have been around since the dawn of PKI implementation, yet traditional admins want to move away from this tried and tested methodology and come up with elaborate solutions for virtualising the Root CA. This is normally because the organisation does not want to buy a separate HSM for the Root CA, but rather use one network based HSM for both online and offline environments. This can be achieved as most network based HSMs have two interfaces where there is no network connectivity between the two. My main issue taken with this approach is twofold, firstly it’s the complexity of the solution, secondly it’s the fact that the CA needs to be connected to some network, private or not, to use its private key material which increases its surface area for attack. You can use a USB based HSM in some virtual environments which support direct USB pass-through (not over the RDP channel as far as I’m aware) but I would ask the question to all PKI administrators, why?

The below diagram is a simple design of the above mentioned solution, there are many variations on this design but nearly all involve connecting the offline CA to its own “private” HSM interface:

Taking Issue

The fundamental question we need to ask ourselves is, what is the problem we are trying to solve in virtualising the offline CA? Is it cost, hardware independence or a love for all things virtual? I’d like to do some debunking here on the virtualisation approach and put forward a case as to why I think the traditional method is still the best.

As mentioned, my main mine gripe is complexity, let’s take a look at what we might need to do to secure our virtual CA such that we can state it has equivalent security to an offline CA.

Firstly, we need to ensure the virtual host has the same security as what our offline CA would give us, most notably the physical host server would need to be protected to the same impact level of a normal offline CA. This would involve housing the virtual host server in a secure rack with physical access controls and have the policies and processes to ensure only designated personnel have access to the CA. All media introduced to the CA would need to be controlled and malware checked. If the CA is hosted on another platform other than Hyper-V, you may also require an administrator workstation to connect in to the host server so you can manage the guest, this workstation would need to be built from a known good image and used exclusively for the CA management. This isn’t completely unachievable but does become unworkable when you need elegant, easy to follow scripted procedures for Key Signing Ceremonies.

Secondly, by the very nature of the CA being virtualised you will need to connect it to a network and a network based HSM. This will require infrastructure and means that the HSM to which the offline and online CAs are connected to would need to be protected to the highest impact level of the all the connected assets. So, in this instance the HSM would need to be stored in the secure rack with the same access control procedures as defined for the Root CA. In the scenario where the Root CA is completely offline we still need to protect the online CA HSM, but we can define a separate security zone as these assets are online and need to be network connectable which provides distinct separation and allows us to split up the management of the two platforms. Remember our Root CA private key is the crown jewels we are trying to protect and we don’t want to unduly expose it by having network connectivity to the area where it is stored from a lower level security zone or to lower level administrators.

All of this network setup and thinking of assets crossing security zones just screams too much complexity to me and seems like over-engineering for a solution which really doesn’t need to be complex. With this being said, I have had customers propose a “simpler” approach to the above problem where the Root CA is stored on their production virtualisation platform where their online CAs are stored, the offline CA is then connected with private virtual switches, host NICs and cabling to the HSM. The root CA would then be taken offline by dismounting the virtual machine when not in use and encrypting the virtual hard disks such that the CA cannot be started. To bring it online a process would exist whereby the disks are decrypted and the virtual machine is mounted temporarily. Again, I would ask why? Why engineer the solution to be this complex? What is the problem you are trying to solve? Don’t make the design of your PKI overly elaborate when it doesn’t need to be.

Now to debunk some of the reasons for the virtualisation design:

Cost

The cost of a USB or PCIe HSM card is very reasonable today, and although I can’t give you exact figures it is somewhere in the region of £2.5K - £5K depending on supplier, vendor and model you go for. This may initially look like a lot of money but when you take into consideration the extra administration overhead you have with the virtual solution, the cost of the extra infrastructure and the pain you will go through to achieve the security you need it’s actually a drop in the ocean. You can buy a cheapish laptop and USB HSM and then store these in a protected safe when it is not in use and achieve a high level of data separation and security for relatively low cost. This will give you a high level of security and confidence in the integrity of your Root CA without the over-engineering of the solution. So, no, I don’t take cost as an argument for the virtualisation of the Root CA.

Hardware Independence

Another argument that people sometimes use is around hardware independence, such that if you had a system failure you could restore the VM to any other physical server with the same hyper-visor technology very easily. The argument is that you aren’t reliant on the same physical hardware for system state restores and you get flexibility in your recovery approach and disentangle yourself from specific hardware components. I would however still have a retort to this in that if you have a robust backup and restore process following a componentised model, like the one I documented in this post, this argument becomes irrelevant. In my opinion the component based restore is elegant and not only allows you to restore your service but also migrate it to a new hardware platform or even a new up level operating system, such as during a Windows Server upgrade. The component based restore on a dedicated physical platform is also cost effective.

Summary

This post ended up being more of a rant than I wanted it to be, but we can sometimes get carried away with designing and implementing IT systems fixing problems that don’t exist and I’m a big believer in keeping things simple. The age old saying of “complexity is the enemy of security” rings true for this scenario. Virtualisation absolutely has a place at centre stage in our environments today, but in my opinion, it does not have a place when it comes to offline CAs. I can speak from experience of being in many Key Signing Ceremonies with third party auditors looking over your shoulder and will say that the last thing you want in this scenario is too much complexity as something is bound to go wrong and cause you lots of pain and questions from the auditor.

Alas, I am a bit of a PKI purist though, some may say a curmudgeon, so my views are slightly skewed. So with all of this being said, assess the needs of your PKI, what assets you are protecting and what level of trust you are placing in the certificates it issues. It may be the case that you don’t need extravagant security controls and protection mechanisms and therefore can ignore all of my advice :)

A good friend of mine, David Wozny, wrote an excellent whitepaper for Thales on Offline CA design principles which can be found here, and he covers off what I have spoken about in this post a lot more succinctly. I encourage you to read his whitepaper and this blog post together before you make any decisions on your offline Root CA implementation.

Public Key Infrastructure is my speciality and today I'd like to share some thoughts on disaster recovery. This blog post is an update to the disaster recovery article posted in 2010 linked here. When working with customers I like to look at disaster recovery by breaking it down into scenarios that could go wrong and how each scenario can be recovered from. Also, there are some mitigations that can be performed in an emergency situation if you absolutely cannot get a CA back online for a period of time, which I would like to touch on. I usually put these into a Backup Matrix so the different scenarios and their specific problems can be addressed easily, and because anything with matrix in the title sounds cool!

This post will use many acronyms throughout and assumes at least an intermediate level of knowledge about PKI. So, without further ado, fasten your seat-belts as this post will be a whopper!

The Matrix

The table below is the backup matrix I was referring to. This is not an absolute authoritative list of everything that can go wrong with a PKI but provides some high level common scenarios. The recovery steps listed are broken down further below with some detailed advice and guidance on what needs to be backed so these recovery processes can be followed.

Backing up a Certificate Authority

So, to the fundamentals first. Before we can restore a certificate authority we need to have a good valid backup. The best way to backup a certificate authority is to use the certutil command line tool, the Swiss army knife of anything PKI! There are three parameters you can use to backup the CA depending on the information you want to capture, they are as follows:

• Certutil –backup è This will backup both the database and private key of the CA

• Certutil –backupDB è As the switch suggests this will backup the database only

• Certutil –backupKey è Again, as the name suggests this will backup the CA private key only

Which one do I use? I hear you ask, well that depends on a couple of things really. Firstly, are you using a Hardware Security Module (which is best practice for a CA)? If so, you will never use the full backup or backup key options as the private key backup and recovery will be specific to the type of HSM you are using. If you aren’t using a HSM, then you can use the other backup options to backup the key, however, once installed the CA private key may only change at very lengthy intervals. This may be when the CA certificate is renewed or otherwise re-keyed due to other factors. Based on this we only need to have a valid backup of the current private key which may be valid on a CA for many years, such as for a root CA. In my opinion it is only needed to backup the private key at these intervals and documentation should exist to either do a “certutil –backupkey” or to export the CA certificate manually using MMC console at said intervals. When the backup of the private key is taken it should have a very secure password configured and safely recorded, such as in a fire safe with limited access. As the private key is the crown jewels of the CA the backup of the key should also be secured and not left lying round on the server or otherwise easily accessible location. This is where the value of HSM’s come in as ordinarily anyone who has administrative access to the CA server can export or backup the private key to a PKCS#12 file and then import onto their own server and impersonate your CA, which is very bad! With a HSM the private key is protected in a hardware device which has special controls in place such that the keys cannot just be exported.

Now we have a good backup of the CA certificate and private key we can look to backup the CA database. As I mentioned already, the easiest and most elegant way to do this is to use the “certutil –backupDB” command line, indeed wrapping this into a batch file and running on a schedule makes the most sense. In addition to the backup of the CA database allowing for CA recovery, backing up also has the added benefit of truncating the CA log files. Some good information on huge CA databases, compaction and truncating log files is detailed in this blog post. Here is a sample script for backing up the CA database that I use, it is very rudimentary but gets the job done:

One thing you will notice about the script is that it creates folders for the last seven days so there are seven backups. Also, the script backs up the CA registry settings, CAPolicy file and the event logs for server. I think the event logs are useful, especially the security log, so that you have an audit trail of the CA that can be archived. I must mention that I always turn the auditing on the CA up to max as well, you never know when you’ll need to have that audit information! Now, I usually configure the script to run every 24 hours so there is a fresh backup every day for the last seven days stored on the data drive of the server. From there other tools can be used to backup the file system of the server for full server recovery, longer term archive and data retention. There are also some new PowerShell cmdlets that can be leveraged to perform the CA backup, so if you’re on a 2012 or later server you can leverage PowerShell instead of the old batch files.

In addition to the backup of the CA database it is a good idea to have copies of the original installation scripts and CAPolicy file. You do have detailed installation documents and configuration management, right? I think the best thing to do in this situation is to have all of the commissioning scripts in a folder on the data drive where the CA database backup is, that way when it comes to restoring the database using the long term file backup solution the original scripts can also be restored as well. I also think it’s a good idea to publish a CRL every 24 hours regardless of the CRL validity period settings and to do this within the backup script. The main reason for this is because in my experience certain certificate policies require you to publish every 24 hours. This also ensures CRL information is up to date, see detail later about how the CDP can be kept up to date with this data as well.

The last thing on backing up the CA I want to mention is around having a record of all certificates that are issued. If you are only backing up the CA every 24 hours then the worst case scenario is that you will have nearly 24 hours’ worth of issued certificates that aren’t included in the backup. For a high volume CA that is pumping out lots of certificates this can be an issue. The certificates themselves will continue to work and the users, devices or services which are using them will function as before, however the big risk is that you can’t revoke those certificates. From a security point of view this is bad as potentially there are many certificates out in the wild that you have lost control of. There is a trick we’ll talk about later that can be used to enable the CA to have knowledge of these lost certificates or if all else fails we can revoke the certificates if we have their serial number. The Microsoft CA has a built in feature we can use to help us record the serial number and base64 encoded value of every certificate that is issued, this is done using the SMTP exit module. The SMTP exit module is not configured by default so this will need configuring during the initial building of the CA.

Detailed information on the SMTP exit module and how to configure it are detailed in the following articles:

http://blogs.technet.com/b/xdot509/archive/2013/06/17/operating-a-pki-smtp-exit-module.aspx

http://social.technet.microsoft.com/wiki/contents/articles/2004.active-directory-certificate-services-smtp-exit-module-for-windows-server-2008-r2-example.aspx

Note in the script that you will need to configure the correct parameters for your SMTP server and emails addresses you wish to use.

Recovery Scenarios

Complete CA Failure

Now we have a full backup of the CA we can look at the different scenarios and what methods we can use to restore the CA to service or at least ensure clients and certificate validations are working. So in our first scenario lets imagine the CA has had a complete failure, this could be anything from a physical server crashing and burning or if virtual a corruption of the virtual hard disk. In this scenario the CA will be unable to issue any new certificates or publish a new CRL. Now, it may appear that being unable to issue new certificates is the most important issue to fix so service can be restored but being unable to publish a CRL is a far more important issue to address. Not having a current valid CRL will have a huge impact on a business where certificates are used heavily. Let’s take an example of an organisation that uses smartcards for Windows logon, if the CA CRL expires a situation will occur where all of their users are unable to logon, I have indeed seen this happen in my career. From a business continuity point of view users not being able to log on to their Windows desktops on a Monday morning is just about the worst thing that could happen. Now, the amount of time you have before this world ending event occurs depends on what your CRL publication interval and its overlap period are. This is essentially your SLA. By way of example, if you have a CRL which has a base validity period of 24 hours and an overlap period of 24 hours you have at a minimum 1 day and at a maximum 2 days to recover your CA. One point to make on this is that if you are using Delta CRLs with a shorter validity period, say 1 hour, then that will be become your minimum time for recovery as Delta CRLs are critical for a PKI authentication.

Ok, now that whole complicated explanation is out of the way how do we fix this and not allow this apocalyptic event to occur. Well there is only one way to fully restore the CA, which is to recover the CA from backup onto a new or rebuilt server. This may take some time depending on your own infrastructure, your level of monitoring and alerting for the CA (check back later for another post on this!) and ability to action the server rebuild. If you cannot fully restore the CA within the SLA period (the CRL validity period) then there is a process to re-sign the CRL on another server and extend its validity so that clients can validate the CRL and carry on with their business. Obviously new certificate issuance and revocations will still be on hold until the CA is recovered but at least the CEO will be able to logon and check his emails.

Full CA Recovery

The following steps can be used to fully recover a CA:

1. Build a new server, either new hardware or virtual, as your CA. It is best practice to use the same server name here, this will save some headache later.

2. Configure server OS to your standard, set IP address and join to the domain.

3. Restore the directory previously backed up to the data drive on the CA, this should include 7 days’ worth of CA database backups, the various certutil outputs, event logs and commissioning scripts.

4. Copy the CAPolicy.inf file back into the %WINDIR% directory.

5. Obtain a copy of the private key form secure storage and make available on the CA, this may be restoring the PFX file if not using a HSM or by configuring the HSM and restoring the private key using the HSM manufacturer’s instructions.

   1. When using a HSM it is usually necessary to re-associate the hardware protected private key with the CA certificate. To do this, the CA certificate will need to be installed into the machine store of the CA (it can be retrieved from the AIA location where it should be published) and the CA private key available in the HSM device.

      Once the certificate and private key are available the following command can be run to join the key to the certificate:

       

You will receive feedback similar to the following:

Note you will need to replace the CSP section with the CSP of the HSM manufacturer and the CA certificate serial number with the serial number of the CA certificate you are restoring.

1. Once the certificate and private key have been restored by either importing the PFX file locally on the server or by using the HSM you should be able to see the certificate in the local machine store with the private key symbol as shown in the following screenshot:

1. If you are running on a 2012 upwards CA you can now install the Active Directory Certificate Services role before we perform the configuration. If you are on a lower level operating system then just install the role as normal and configure the installation as per the instructions in the next step.

2. Now we can perform a recovery installation of the CA following these steps, note the screenshots are from a 2012 R2 server but the options are the same across any CA version:

   1. Select Enterprise CA or Standalone CA depending on which CA you are restoring.

   2. Select Root CA  or Subordinate CA depending on which CA you are restoring:

   3. Select Use existing private key and then the Select a certificate and use its associated private key option:

   4. The certificate you imported earlier should be listed in the Certificates dialog box, ensure this is selected and click Next

      Note: If you are using a HSM there may be a need to check the “Allow administrator interaction…” check box. Ensure this setting is configured if required by the HSM vendor.

   5. Configure you database and log location, this can be the same as before:

   6. Observe the confirmation of the settings and click Configure (or Install on a non-2012 server):

   7. Click Close to finish the configuration:

3. Once the CA role has been installed and configured the CA database can be restored, this can be done through the CA GUI or via the command line by running the following commands:

    

Note that the restore path above will be correct based on the sample script that is detailed above. Also note the –f parameter to force overwrite the existing database created during the install.

1. Now the CA server, private key and database have all been restored the original install script can be re-run to configure the CA. This will ensure the correct CDP and AIA paths are configured and CRL lifetimes as per the original install. If you are in doubt of the original config then the CA registry export in the backup folder can be used to rebuild the configuration of the CA. Any other configuration should now be applied to the server such as permissions on the CA, certificate managers etc.

2. All certificate templates which were previously published to the CA should be re-published. The list of these templates will be in the ca-templates.txt file located in the backup folder.

3. Any scheduled tasks should also be configured for the monitoring of the CA and copying og the CRL as per the original install.

4. The CA will now be fully operational and should be validated to ensure it is functioning correctly. You can test a certificate enrolment and revocation and then publication of the CRL to ensure this is working.

5. The last thing to do is to replay any certificates into the database that have been issued since the last backup. As we touched on earlier we can get the details of the certificates by filtering the inbox where the SMTP exit module sends the emails to. This should be filtered to only show emails which have been received between the last backup date and the current time. We can then look at all emails which contain details of the issued certificates. The following screenshot shows a sample of this from my lab:

   As you can see, the email contains certificate details but pay particular attention to the base64 encoded certificate, which can be taken and pasted into notepad to re-construct the whole certificate. Follow these steps to replay the certificates to the database:

   1. Save the base64 encoded certificates from each of the emails to a *cer file in a folder on the CA. You should have a .cer file for every certificate you wish to restore.

   2. Before we can replay any certificates into the database we need to configure the CA to allow this by running the following command:

       

1. Run the following command to add the certificate into the database:

    

Where <certificate file> is one of the saved files. You could script this so it loops through all of the certificates in the folder one by one and adds them to the database if you have many.

Once all of the above steps have been completed your CA should be fully recovered and up to date with all issued certificates. The above procedure can also be adapted to migrate a Certificate Authority to another server, say during an upgrade to a newer platform, the concept is essentially the same. There are a few differences in the exact steps the full details of which can be found here. The process can also be used to recover the CA to a test environment for validating your disaster recovery plans and backups.

Manual CRL Resign

As you can see from the steps above a full CA recovery can be a lengthy process, particularly if you have not been alerted to its failure in a reasonable time frame. As we mentioned earlier this may mean that the CA cannot be recovered before the CRL is due to expire which would have dire consequences to any applications or services relying on that CA’s certificates. To mitigate this a manual re-sign of the CRL can be done to extend the validity of the CRL for a short time period while the full recovery above is performed. The following steps go through a manual CRL resign:

1. Log on to a server in your environment, preferably another CA, or you could use an up to date client operating system such as Windows 8.

2. Obtain a copy of the private key and make available on the machine, as detailed above this may be restoring the PFX file if not using a HSM or by configuring the HSM (which is why another CA is preferred) and restoring the private key using the HSM manufacturer’s instructions. I won’t go into the details of this as they are above in the Full CA Recovery section, but you should have the CA certificate and private key installed with the certificate showing the private key symbol as seen in the screenshot further above.

3. Obtain the latest CRL from the CDP and copy to a temporary directory on the machine, rename the CRL by adding a –old after the file name.

4. Extend the CRL lifetime by running the following command:

    

Where 0:XX is the amount of hours you want to add to the CRLs validity, such as 0:48 for adding 48 hours. You can specify which ever time period you desire here, but don’t go crazy and only specify a time period long enough to be able to perform a full recovery. It is tempting to add many days here, but remember that the CRL may then potentially get cached at the client side for that period of time and the client may not then retrieve an up to date CRL for that extended period.

Also note that the old file is specified first and then the newly signed file. The newly signed file should have the name that is referenced in the CDP.

1. The newly signed CRL should then be published to the CDP, this may include a web CDP and LDAP CDP. For the web CDP, copy the file over to the virtual directory and overwrite the existing file, for the LDAP CDP run the following command:

    

1. Once the CRL has been published then the CDPs should be checked by using the Enterprise PKI console, you should see the status as OK  for all CDPs, similar to the below screenshot:

Another point to make here, although slightly off topic, is that the above certutil command can be used to add entries to the CRL. So if you haven’t restored the CA fully or you are having some issues and you need to revoke a certificate in a hurry (maybe an employee has gone rogue?) you can manually add a serial number to the CRL by running the following command:

Where <serial number> represents the serial number of the certificate you wish to add to the revocation list such as “42 00 00 00 05 06 0a 1a 29 6d 40 a1 3e 00 00 00 00 00 05”

Loss of Access to Private Key

Let’s now imagine a scenario where the CA loses access to its private key only. This could be due to a HSM failure or even accidental deletion of a software based key from the local Crypto folder by an administrator. In this instance the private key should be restored following step 5 in the Full Recovery section above. Once restored the service should be restarted and the CA should then function correctly.

Database corruption or other database issue.

Another scenario that could happen is if there is database corruption for some reason, maybe due to a disk error or something similar. The process for recovery or mitigation is similar to the steps involved for a full recovery. Ultimately a recovery of the CA database will need to be done following steps 8 and 11 above in the Full Recovery section. Once the database has been restored then the service should be started and in theory will be good to go.

If it will take some time to recover the database then the procedure for resigning the CRL can be performed to extend the CRL lifetime.

CDP is Unavailable

When designing CRL distribution points you should ensure that redundancy is built into the design to cope with any failure at any point in the infrastructure. You should typically have more than one CDP listed and each CDP should be accessible from the clients you wish to serve, e.g. if you are providing services to remote clients then the CDP should be externally accessible. I normally advocate having a web CDP as the first one in the list due to its ubiquity amongst client types, including network devices and other non-Windows clients. This ensure that clients of all flavours will be able to retrieve a CRL from the first CDP in the list. Remember that with PKI the first CDP in the list is tried by the relying party, and if this is unavailable the second CDP is tried using the retry algorithm, this is explained in great detail here, although each device type may be different. So, having a web CDP first should minimise timeouts while connecting to the CDP. The downside to using a web CDP is that the server on which it is hosted needs to be redundant using something such as Network Load Balancing, or at a minimum DNS round robin. This adds complexity and potential cost to the web service when it may only be serving files of a few kilobytes. It would be good practice to configure the CDP on another highly available web property that you own which is already configured for load balancing and monitored for up time.

Another area of concern for a web CDP, probably the biggest area for failure, is in the process used for copying the CRL from the CA over to the web server. Generally the CA will be configured to publish the CRL to the local file system and then a scheduled task of some sort used to copy the file over to the server. You can use any mechanism you like here for copying the file, maybe a PowerShell script, a VBScript or a good old batch file using robocopy. The way I like to do this is to create a task attached to Event ID 4872 in the security log. This audit event is generated in the security log when you have CA auditing switched on within the CA console and the “Audit Object Access” local security setting configured. I highly recommend you configure these settings for your CA. More information on the CA audit events can be found here. The steps for creating this task are listed below:

1. Open the security log in event viewer and filter for event ID 4872 as per the following screenshot:

2. In the results select one of the audit events for the CRL publication, right click this and select Attach Task To This Event…

3. Give the task a name and description how you like and click Next.

4. Click Next again.

5. Select to Start a program and click Next.

6. Select the script you wish to run, again this can be any copy script you desire, and click Next.

7. Click Finish to create the task.

Now any time the CA publishes a CRL, either automatically or manually, the copy script will run and push that CRL to the web CDP. This mechanism isn’t infallible but generally works quite well. You may wish to add your own monitoring to make sure the script is running, maybe by sending a mail on CRL copy or by writing a log file. This mechanism also has the added bonus that the CRL file on the CDP will always be the latest one instead of waiting for some other arbitrary schedule.

The other type of CDP you can configure is LDAP based, where a Microsoft CA will store CRLs in Active Directory. LDAP CDPs may not be supported by all devices, hence my reasoning for adding them second in the list, but they provide redundancy “out of the box”. By being stored in AD they are replicated to all domain controllers in the forest as they are stored in the configuration partition. AD provides inherent redundancy via its multi-master model but also allows clients to use AD sites to connect to their closet domain controller. A Microsoft CA can also publish its CRL into AD automatically on CRL publication, so no extra scripts or configuration are required.

To summarise CDP failure, it is vitally important to ensure you have multiple CDPs configured and that with web CDPs some form of load balancing is implemented. Also, a robust copy mechanism should be used to ensure the web CDP always has the latest CRL available. LDAP CDPs have redundancy and availability by virtue of being stored in AD.

Custom templates removed from directory or lost due to other AD related failure

The last disaster recovery point I want to cover is around certificate templates. Imagine that an administrator has accidently deleted all certificate templates from AD or something has gone wrong with AD whereby the templates are unusable. There isn’t a built in way to backup and restore those templates apart from using the AD tools. This is a bit heavy handed to just restore the certificate templates. To help with managing this I’ve created a PowerShell script which will back up the certificate templates to an XML file which can later be used to re-create all of the certificate templates. The script can be found here. This script uses the Certificate Enrolment Web Services and Certificate Enrolment Policy Web Services roles, as the API needed to dump the templates out this way was added with this service. As far as I’m aware there isn’t any other supported mechanism to dump the certificate template objects out of AD. It should be noted that the CEP and CES services should be installed and you the script needs to be run on the box where this service is. More detail on the script can be found on the script centre posting.

Once you have a backup of your certificate templates the script can be used to re-create the exported templates within any Active Directory environment. All elements of the exported templates will be re-created apart from security permissions, these will need to be manually re-added.

Other Mitigations

Using the backup and restore information above along with the backup matrix you should be well equipped to create your own disaster recovery documentation and approach to recovering you CA infrastructure. However you may still be in a predicament whereby you can’t execute a recovery for some reason, maybe all your backups have been lost, stolen or otherwise corrupt. Well there are still options to allowing users, computers and services to carry on functioning when a new CRL cannot be issued. You can switch off certificate revocation checking.

NOTE: I do not advocate using this approach unless you absolutely have no choice in the matter, as disabling revocation checking is a huge security risk. Follow the below advice at your own risk!

Revocation checking is down to each individual application and will need to be configured on a per-app basis depending on what application you need to restore functionality too. As an example, if you are using smartcards for logon then you can disable revocation checking at the domain controller to allow user certificates to carry on being validated even when the CRL has expired. As noted, this is a big risk but maybe your only choice. Other applications and services will have their own method for disabling CRL checking and you should look this information up or if need be contact the application vendor if it is a non-Microsoft application, service or device.

Summary

To summarise this monster post and close out I’d like to leave you with a few key points:

1. Having a robust backup strategy for your PKI is essential and this should be documented. You can use the approach detailed above to help build this out.

2. There is great value in understanding the different disaster scenarios and ensuring you have a restoration strategy for them. Using something like the backup matrix and referencing the particular restore process would be beneficial.

3. There are some techniques you can use to get yourself up and running in an emergency. Re-singing Re-signing the CRL is preferable but as an absolute last resort you can disable revocation checking.