Certificate rotation
You may need to rotate the certificates used by your Dynamics 365 Finance + Operations (on-premises) environment as they approach their expiration date. In this article, you learn how to replace the existing certificates and update the references within the environment to use the new certificates.
Warning
The certificate rotation process should be initiated well before the certificates expire. This is very important for the Data Encryption certificate, which could cause data loss for encrypted fields. For more information, see After certificate rotation.
Old certificates must remain in place until the certificate rotation process is complete, removing them in advance causes the rotation process to fail.
The certificate rotation process should not be carried out on Service Fabric clusters running 7.0.x and 7.1.x.
Upgrade your Service Fabric cluster to 7.2.x or later before attempting certificate rotation.
Preparation steps
Update your infrastructure scripts by following the steps in Update your Infrastructure Scripts.
In the ConfigTemplate.xml file, configure certificates as you require. Follow the steps in Configure certificates. Specifically, follow these steps.
# Only run the first command if you have not generated the templates yet. .\New-ADCSCertificates.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -CreateTemplates .\New-ADCSCertificates.ps1 -ConfigurationFilePath .\ConfigTemplate.xmlNote
The AD CS scripts need to run on a domain controller, or a Windows Server computer with Remote Server Admin Tools installed.
The AD CS functionality is only available with Infrastructure scripts release 2.7.0 and later.
Alternatively, if you want to continue to use self-signed certificates, run the following command.
# Create self-signed certs .\New-SelfSignedCertificates.ps1 -ConfigurationFilePath .\ConfigTemplate.xmlWarning
Self-signed certificates should never be used in production environments. If you're using publicly trusted certificates, manually update the values of those certificates in the ConfigTemplate.xml file.
After you've generated the certificates, run the following command.
# Exports certificates into a directory VMs\<VMName>. All certs are written to the infrastructure\Certs folder. .\Export-Certificates.ps1 -ConfigurationFilePath .\ConfigTemplate.xmlContinue to set up VMs. Here are the specific steps that are required for this process:
Export the scripts that must be run on each VM.
# Exports the script files to be executed on each VM into a directory VMs\<VMName>. .\Export-Scripts.ps1 -ConfigurationFilePath .\ConfigTemplate.xmlCopy the contents of each
infrastructure\VMs<VMName>folder into the corresponding VM (if remoting scripts are used, they automatically copy the content to the target VMs), and then run the following script. Perform this step as an administrator.# If remoting, only execute # .\Configure-PreReqs-AllVMs.ps1 -MSIFilePath <share folder path of the MSIs> -ConfigurationFilePath .\ConfigTemplate.xml -ForcePushLBDScripts .\Configure-PreReqs.ps1Run the following script to ensure that all prerequisites are completed. Perform this step as an administrator.
# If remoting, only execute # .\Complete-PreReqs-AllVMs.ps1 -ConfigurationFilePath .\ConfigTemplate.xml .\Complete-PreReqs.ps1Run the following script to validate the VM setup.
# If remoting, only execute # .\Test-D365FOConfiguration-AllVMs.ps1 -ConfigurationFilePath .\ConfigTemplate.xml .\Test-D365FOConfiguration.ps1
Run the following PowerShell command so that you have values that can be used in Lifecycle Services later. For more information, see Deploy your on-premises environment from Lifecycle Services.
.\Get-DeploymentSettings.ps1 -ConfigurationFilePath .\ConfigTemplate.xml
Activate new certificates within Service Fabric cluster
To make the certificate rotation process easier, Microsoft highly recommends that you use certificate common names (subject name) instead of thumbprints for your Service Fabric standalone cluster configuration. If you have an existing cluster and want to migrate from using thumbprints to using certificate common names, follow the steps in Appendix B later in this article.
Service Fabric cluster with certificate common names
Service Fabric with certificates that aren't expired
No further action is required on the Service Fabric cluster. Service Fabric automatically detects the new certificates. You should proceed with Update the LocalAgent certificates.
If you've changed the certificate common name, you must upgrade your Service Fabric cluster configuration.
Run the following script to generate an updated cluster configuration file.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -UpdateCommonNamesNote
If the issuers have also changed, use the following command instead.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -UpdateCommonNames -UpdateIssuersApply the updated configuration to your Service Fabric cluster by using the information in Appendix A later in this article.
Service Fabric with certificates that are expired
If your cluster isn't available after 10 minutes from when you finished provisioning the new certificates to all nodes, consider restarting the nodes where the Service Fabric service isn't started.
If you have changed the certificate common name (subject name), then the Service Fabric cluster won't start up. If you can't generate new certificates with the previous common name, you need to clean up and recreate the cluster.
Service Fabric with restricted certificate issuers
If the following sections are defined for your cluster configuration, the allowed certificate issuers are restricted.
"ClusterCertificateIssuerStores": [
{
"IssuerCommonName": "sample-ca",
"X509StoreNames": "Root"
}
],
"ServerCertificateIssuerStores": [
{
"IssuerCommonName": "sample-ca",
"X509StoreNames": "Root"
}
],
"ClientCertificateIssuerStores": [
{
"IssuerCommonName": "sample-ca",
"X509StoreNames": "Root"
}
],
In this case, if the issuer of your new certificates differs from what is defined in these configurations, you must go through a cluster configuration upgrade to add the new issuers.
If you have to update the list of issuers, you must do the update while the existing certificates are still valid.
Run the following script to generate an updated cluster configuration file.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -UpdateIssuersApply the updated configuration to your Service Fabric cluster by using the information in Appendix A later in this article.
Service Fabric cluster defined with certificate thumbprints
Service Fabric with certificates that aren't expired
Run the following script from a node that belongs to the Service Fabric cluster to generate an updated cluster configuration file.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -UpdateThumbprintsApply the updated configuration to your Service Fabric cluster by using the information in Appendix A later in this article.
Service Fabric with or without expired certificates (cluster not accessible)
Continue this process following the steps in Troubleshoot on-premises deployments.
Update the LocalAgent certificates
You must reinstall the LocalAgent in the following situations:
- You changed the Service Fabric cluster/server certificate.
- You changed the Service Fabric client certificate.
- You changed the LocalAgent certificate.
Update your current localagent-config.json file by replacing the serverCertThumbprint and clientCertThumbprint values with the new thumbprints.
{ "serviceFabric": { "connectionEndpoint": "192.168.8.22:19000", "clusterId": "Orch", "certificateSettings": { "serverCertThumbprint": "New server thumbprint(Star/SF)", "clientCertThumbprint": "New client thumbprint" } },Run the following PowerShell command on one of the Orchestrator nodes.
.\LocalAgentCLI.exe Cleanup <path of localagent-config.json>Run the following PowerShell command and note the new LocalAgent thumbprint.
.\Get-AgentConfiguration.ps1 -ConfigurationFilePath .\ConfigTemplate.xmlFollow the steps in Configure Lifecycle Services connectivity for the tenant.
Note
If you receive the error Update to existing credential with KeyId '<key>' is not allowed, follow the instructions in Error: "Updates to existing credential with KeyId '<key>' is not allowed".
Continue with Configure a connector and install an on-premises local agent, specifically the following changes:
- Client certificate thumbprint
- Server certificate thumbprint
- Tenant service principle certificate thumbprint
Important
Do not create a new connector in Lifecycle Services. Update the configuration of your existing connector and download the settings file again.
Update your current deployment configuration
Note
This is only necessary with local agent 3.2.2 and older. If you are using local agent 3.2.3 or later, you can skip this step.
Because you've updated your certificates, the configuration file that is present in your environment is outdated and must be manually updated. Otherwise, the clean-up job may fail. This manual update must be done just this one time.
Open the configuration file 'config.json' on your agent file share. This is in a share similar to the following: \\fileserver\agent\wp\environmentID\StandaloneSetup-123456. You can find the location of this file by running the following SQL statement on the orchestrator database.
select Location from DeploymentInstanceArtifact where AssetId='config.json' and DeploymentInstanceId = 'LCSENVIRONMENTID'Note
Replace LCSENVIRONMENTID with the ID of your environment. You can obtain this ID from the page for your environment in Lifecycle Services (this is the GUID value associated with your environment).
The beginning of the file should resemble the following example.
{ "serviceFabric": { "connectionEndpoint": "192.168.8.22:19000", "clusterId": "Orch", "certificateSettings": { "serverCertThumbprint": "Old server thumbprint(Star/SF)", "clientCertThumbprint": "Old client thumbprint" } },Replace the serverCertThumprint and clientCertThumbprint values with the new thumbprints.
{ "serviceFabric": { "connectionEndpoint": "192.168.8.22:19000", "clusterId": "Orch", "certificateSettings": { "serverCertThumbprint": "New server thumbprint(Star/SF)", "clientCertThumbprint": "New client thumbprint" } },Save and close the file. Remember to close any programs that are accessing this network location. Otherwise, the cleanup process might fail.
Rotate Credentials.json
If you've generated a new axdataencipherment certificate, you must re-encrypt the Credentials.json file.
Note
Make sure that you run the script from an Application Object Server (AOS) node.
.\Configure-CredentialsJson.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -Action Rotate
Alternatively, if you also want to rotate the existing credentials, follow these steps.
Decrypt the Credentials.json file.
.\Configure-CredentialsJson.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -Action DecryptOpen the Credentials.json file, and update any credentials that you want to update.
Re-encrypt the Credentials.json file.
.\Configure-CredentialsJson.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -Action Encrypt
Update deployment settings in Lifecycle Services
Note
The Client, Data Signing, and Encipherment certificates will only be replaced.
Before you continue, you need to make a backup of the local Dynamics database.
In Lifecycle Services, select the "Full Details" link for the environment where you want to change the certificates.
Select Maintain and then select Update Settings.
Change the thumbprints to the new thumbprints that you previously configured. You can find them in the ConfigTemplate.xml file in the InfrastructureScripts folder.
Select Prepare.
After downloading and preparation is complete, the Update environment button displays.
Select Update environment to start updating your environment.
During the update, the environment is unavailable.
After the environment is successfully updated with the new certificates, you can view the new thumbprints in Service Fabric Cluster Explorer. The names of the thumbprints in Service Fabric Explorer might differ from the names in Lifecycle Services. However, the values should be the same.
Here is an example of how the name of the same thumbprint might differ.
Update other certificates as needed
Always check if the SQL server certificate has expired. For more information, see Set up SQL Server.
Check to be sure that the Active Directory Federation Service (ADFS) certificate hasn't expired.
Clean up old Service Fabric certificates
This procedure should be completed after a successful certificate rotation or before the next certificate rotation.
Run the following script to generate an updated cluster configuration file.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -RemoveOldThumbprintsApply the updated configuration to your Service Fabric cluster by using the information in Appendix A later in this article.
After certificate rotation
Data encryption certificate
This certificate is used to encrypt data that is stored in the database. By default, specific fields are encrypted by using this certificate. You can check those fields in Document the values of encrypted fields. However, you can use our API to encrypt other fields as you require.
In Platform update 33 and later, the Encrypted data rotation system job batch job uses the newly rotated certificate to re-encrypt data. This batch job crawls through your data to re-encrypt all the encrypted data by using the new certificate. The job runs for two hours each day until all the data has been re-encrypted. To enable the batch job, you must enable a flight and a configuration key. Run the following commands against your business database (for example, AXDB).
IF (EXISTS(SELECT * FROM SYSFLIGHTING WHERE [FLIGHTNAME] = 'EnableEncryptedDataCrawlerRotationTask'))
UPDATE SYSFLIGHTING SET [ENABLED] = 1 WHERE [FLIGHTNAME] = 'EnableEncryptedDataCrawlerRotationTask'
ELSE
INSERT INTO SYSFLIGHTING ([FLIGHTNAME],[ENABLED],[FLIGHTSERVICEID]) VALUES ('EnableEncryptedDataCrawlerRotationTask', 1, 0)
IF (EXISTS(SELECT * FROM SECURITYCONFIG WHERE [KEY_] = 'EnableEncryptedDataRotation'))
UPDATE SECURITYCONFIG SET [VALUE] = 'True' WHERE [KEY_] = 'EnableEncryptedDataRotation'
ELSE
INSERT INTO SECURITYCONFIG ([KEY_], [VALUE]) VALUES ('EnableEncryptedDataRotation', 'True')
After the commands have been run, restart your AOS nodes from Service Fabric Explorer. AOS detects the new configuration and schedules the batch job to run during off hours. After the batch job has been created, the schedule can be modified from the user interface.
Warning
Make sure that the old data encryption certificate isn't removed before all encrypted data has been re-encrypted, and that it hasn't expired. Otherwise, data might be lost.
Appendix A
After you generate the updated Service Fabric cluster configuration, run the following PowerShell commands to apply the upgrade to your Service Fabric cluster.
# Connect to the Service Fabric Cluster
Connect-ServiceFabricCluster
# Get path of ClusterConfig.json for following command
# Note that after running the following command, you need to manually cancel using the red button (Stop Operation) in Windows PowerShell ISE or Ctrl+C in Windows PowerShell. Otherwise, you receive the following notification, "Start-ServiceFabricClusterConfigurationUpgrade : Operation timed out.". Be aware that the upgrade will proceed.
Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath ClusterConfig.json
# If you are using a single Microsoft SQL Server Reporting Services node, use UpgradeReplicaSetCheckTimeout to skip PreUpgradeSafetyCheck check, otherwise it will timeout
Update-ServiceFabricClusterUpgrade -UpgradeReplicaSetCheckTimeoutSec 30
# To monitor the status of the upgrade, run the following and note UpgradeState and UpgradeReplicaSetCheckTimeout
Get-ServiceFabricClusterUpgrade
# While monitoring the status of the upgrade, if UpgradeReplicaSetCheckTimeout was reset to the default (example 49710.06:28:15), run the following command again
Update-ServiceFabricClusterUpgrade -UpgradeReplicaSetCheckTimeoutSec 30
# When UpgradeState shows RollingForwardCompleted, the upgrade is finished
Note
You might receive the following error message: "Upgrading from two different certificates to two different certificates isn't allowed." This message indicates that you didn't clean up old Service Fabric certificates during the previous certificate rotation exercise. In this case, see the Clean up old Service Fabric certificates section earlier in this article, and then repeat the steps in this section.
Appendix B
Using certificate common names instead of thumbprints to describe your Service Fabric cluster configuration eases future certificate rotation operations as the Service Fabric cluster automatically switches to using new certificates once they're available in the machine. Service Fabric won't accept any certificate however, the certificate that is provided must match the subject name that is defined in the Service Fabric cluster. Additionally, the issuer of the certificate must match the issuer that is also specified in the configuration. For more information on how Service Fabric uses common names, see Common name-based certificate validation declarations. For more information on how to secure standalone Service Fabric clusters Secure a standalone cluster on Windows by using X.509 certificates
Run the following script to generate an updated cluster configuration file.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -UpgradeToCommonNamesNote
In some cases, customers may choose to not restrict the issuer of the certificates in the Service Fabric cluster configuration. While this isn't recommended, it can be achieved by using the following command.
.\Update-SFClusterConfig.ps1 -ConfigurationFilePath .\ConfigTemplate.xml -UpgradeToCommonNames -DoNotRestrictCertificateIssuersApply the updated configuration to your Service Fabric cluster by using the information in Appendix A.