CM HTTPS 403 Errors

Question

CM HTTPS 403 Errors

Fred Eric S 51

Hey Everyone,

this is an issue that's been bugging me in one specific environment and we've ruled out pretty much everything else (network, firewall). This isn't my first time setting up HTTPS in a ConfigMgr Site and I've never had issues with this until now. We're on CM 2107 + HF, it's a small environment, so there's just one server (Windows Server 2019) with everything on it. We've created certificate templates according to the standard documentation and issued certificates based on these (client cert, webserver cert, winpe cert). When we switch to HTTPS, the clients pick up their certificate and try to establish communications with the Primary Site, however while doing so and checking the server certificate, the connection is rejected with a 403 error - here's an excerpt from LocationServices.log on a client:

Certificate Issuer 1 [CN=Company-ROOT-CA]  
Based on Certificate Issuer 'CN=Company-ROOT-CA' found Certificate [Thumbprint BF24B51A913520E720C5695B29F79C21E8ADB6D4] issued to 'ClientName.company.domain.de'  
Begin validation of Certificate [Thumbprint BF24B51A913520E720C5695B29F79C21E8ADB6D4] issued to 'ClientName.company.domain.de'  
Completed validation of Certificate [Thumbprint BF24B51A913520E720C5695B29F79C21E8ADB6D4] issued to 'ClientName.company.domain.de'  
Completed searching client certificates based on Certificate Issuers  
Begin to select client certificate  
Begin validation of Certificate [Thumbprint BF24B51A913520E720C5695B29F79C21E8ADB6D4] issued to 'ClientName.company.domain.de'  
Completed validation of Certificate [Thumbprint BF24B51A913520E720C5695B29F79C21E8ADB6D4] issued to 'ClientName.company.domain.de'  
>>> Client selected the PKI Certificate [Thumbprint BF24B51A913520E720C5695B29F79C21E8ADB6D4] issued to 'ClientName.company.domain.de'  
[...]  
Getting CCM Token from STS server 'PrimarySite.company.domain.de'  
Getting CCM Token from https://PrimarySite.company.domain.de/CCM_STS  
Cached encrypted token for 'S-1-5-18'. Will expire at '01/14/2022 19:10:21'  
Begin validation of Certificate [Thumbprint 31F4A2CE105E232D1CD6F5FE3B66D837C361A2EA] issued to 'PrimarySite.company.domain.de'  
Completed validation of Certificate [Thumbprint 31F4A2CE105E232D1CD6F5FE3B66D837C361A2EA] issued to 'PrimarySite.company.domain.de'  
[CCMHTTP] ERROR: URL=https://PrimarySite.company.domain.de/SMS_MP/.sms_aut?SMSTRC, Port=443, Options=63, Code=0, Text=CCM_E_BAD_HTTP_STATUS_CODE  
[CCMHTTP] ERROR INFO: StatusCode=403 StatusText=Forbidden  
Successfully queued event on HTTP/HTTPS failure for server 'PrimarySite.company.domain.de'.

Checking ClientIDManagerStartup.log, there are no errors. When I check on the Primary Site, I can see the following in the IIS Logs:

2022-01-14 10:13:23 10.10.20.XXX CCM_POST /ccm_system_windowsauth/request - 443 - 10.70.144.XXX ccmhttp - 403 16 2148204809 1434 22

Any communication attempt from the client is rejected with a 403.16 error.

MPSetup.log shows no errors, so I can only assume the switch from HTTP to HTTPS was successfull. However, checking MPControl.log, I can see similar errors as above:

Successfully performed Management Point availability check against local computer.  
SSL is enabled.  
CRL Checking is also enabled.  
Using thread token for request  
Call to HttpSendRequestSync succeeded for port 443 with status code 200, text: OK  
Http test request succeeded.  
STATMSG: ID=5465 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_MP_CONTROL_MANAGER" SYS=PrimarySite.Company.Domain.DE SITE=GUV PID=12992 TID=7276 GMTDATE=Fr Jan 14 10:08:42.706 2022 ISTR0="" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0 LE=0X0  
Successfully performed User Service availability check against local computer for /CMUserService_WindowsAuth/applicationviewservice.asmx.  
Applied D:P(A;CIOI;GA;;;SY)(A;CIOI;GA;;;BA)(A;CIOI;GR;;;LS)(A;CIOI;GR;;;S-1-5-17) to folder E:\Program Files\Microsoft Configuration Manager\Client  
SSL is enabled.  
Client authentication is also enabled.  
CRL Checking is also enabled.  
Machine name is 'PrimarySite.Company.Domain.de'.  
Begin validation of Certificate [Thumbprint c221ff4540da1690521b5984e90a08d47be9a049] issued to 'PrimarySite.Company.Domain.de'  
Certificate doesn't have "SSL Client Authentication" capabilities.  
Completed validation of Certificate [Thumbprint c221ff4540da1690521b5984e90a08d47be9a049] issued to 'PrimarySite.Company.Domain.de'  
Skipping this certificate which is not valid for ConfigMgr usage.	SMS_MP_CONTROL_MANAGER	14.01.2022 11:13:42	7276 (0x1C6C)  
[...]  
Begin validation of Certificate [Thumbprint 31f4a2ce105e232d1cd6f5fe3b66d837c361a2ea] issued to 'PrimarySite.Company.Domain.de'  
Certificate has "SSL Client Authentication" capability.  
Completed validation of Certificate [Thumbprint 31f4a2ce105e232d1cd6f5fe3b66d837c361a2ea] issued to 'PrimarySite.Company.Domain.de'  
>>> Selected Certificate [Thumbprint 31f4a2ce105e232d1cd6f5fe3b66d837c361a2ea] issued to 'PrimarySite.Company.Domain.de' for HTTPS Client Authentication  
Call to HttpSendRequestSync failed for port 443 with status code 403, text: Forbidden

Here, the requests constantly switch between 200 and 403.

Based on the error messages, I did a search and came across this article: https://learn.microsoft.com/en-US/troubleshoot/developer/webapps/iis/www-authentication-authorization/errors-403-7-reject-client-certificate-rquest

However, there are no non-self-signed certificates in the trusted root store and no root certificates in the intermediate certificate authorities store. The Root CA cert and the Issuing CA cert are all in the appropriate stores. We've checked and re-checked the cert templates, I made sure the certs are using RSA, not ECDSA, checked the certs common name, subject alternative name, key usage, everything we could think of, doublechecked against the documentation - we couldn't find anything out of the ordinary or different to functioning sites.

At this point I have no clue where to look next. Maybe someone here can spot something obvious that I'm missing?

Cheers,

Fred

Rita Hu -MSFT 9,661 Reputation points

2022-01-20T07:20:49.57+00:00

@Fred Eric S
Thanks for posting on Q&A.

According to the MPControl.log, the CRL Checking is enabled. I found the following information and I want to share with you:
According to the link, Because the location of the CRL is added to a certificate when a CA issues it, make sure that you plan for the CRL before you deploy any PKI certificates that Configuration Manager uses.

In addition, the clients failing to connect to servers if they can't locate the CRL.

Please follow the link to disable the CRL checking first. The link is not from MS, just for your reference. Run the netsh http show sslcert command on the PowerShell to check whether the Verify Client Certificate Revocation : Enabled.

Then we could add the following registry value to disable the CRL checking:
REGISTRY : HKLM\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo
DWORD : DefaultSslCertCheckMode Value : 1
Note that remember to restart the computer for the operation to take effect.

Last we could check the mpcontrol.log to review the result.

Please feedback with me if there are any updates of the case.

Regards,
Rita

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

4 answers

Your answer

Rita Hu -MSFT 9,661 Reputation points

2022-01-20T07:20:49.57+00:00

@Fred Eric S
Thanks for posting on Q&A.

According to the MPControl.log, the CRL Checking is enabled. I found the following information and I want to share with you:
According to the link, Because the location of the CRL is added to a certificate when a CA issues it, make sure that you plan for the CRL before you deploy any PKI certificates that Configuration Manager uses.

In addition, the clients failing to connect to servers if they can't locate the CRL.

Please follow the link to disable the CRL checking first. The link is not from MS, just for your reference. Run the netsh http show sslcert command on the PowerShell to check whether the Verify Client Certificate Revocation : Enabled.

Then we could add the following registry value to disable the CRL checking:
REGISTRY : HKLM\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo
DWORD : DefaultSslCertCheckMode Value : 1
Note that remember to restart the computer for the operation to take effect.

Last we could check the mpcontrol.log to review the result.

Please feedback with me if there are any updates of the case.

Regards,
Rita

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 1

Hey everyone!

We finally got it! I slowly but surely started thinking, that this was not a CM nor a PKI/Cert problem, but rather a problem caused elsewhere - and I was right. Although all error messages point to something being wrong with certificates or client registration, the cause was something else. The golden rule applied:

It's always the network
If it's not the network, it's the PKI
If it's not the PKI, go back to No.1

It was a network related issue. The firewall team did not open specific routes and ports between the client and server subnets, which resulted in the aforementioned errors. I will never again trust firewall teams unless they show me they've done the requested work XD

Cheers,

Fred

Answer 2

Rahul Jindal [MVP] 10,911 MVP

I have seen such log entries on a working client as well. What is in ccmmessaging? Also, have you tested anything to check if the clients are working or not?

Answer 3

Fred Eric S 51

Thank you for the suggestions. I'll look into the logs in question and check the CRLs again. This might take a couple of days, but I'll be sure to get back to this thread with an answer.

Cheers,

Fred

Rita Hu -MSFT 9,661 Reputation points

2022-01-21T02:57:37.277+00:00

Thanks for your response, Fred.

Looking forward for your feedback.

Have a great day.

Rita
Rita Hu -MSFT 9,661 Reputation points

2022-01-26T08:29:35.41+00:00

Hello Fred,

Is there any updates? Any questions or updates will be appreciated. I'm waiting here and looking forward for your feedback :)

Have a good day.

Regards,
Rita
Fred Eric S 51 Reputation points

2022-01-31T12:23:38.827+00:00

Hi Rita,

sorry for the late reply - I'll be looking into the problem this week and give feedback by end of the week.

Thanks for your patience!

Cheers,

Fred

Answer 4

Okay, soooo...

I proceeded as suggested above and deactivated the CRL checking in IIS. This, sadly, did not change anything. I also deactivated CRL checking in ConfigMgr, with the same result. When copying the CRLs from the certs and manually trying to reach them, everything is fine - Edge offers the CRL as a download. Also checked CRL validity, which is ok.

I checked the Client Certificate and the Server Certificate again and could not find anything out of the ordinary - V3, SHA256, RSA (2048), Subject, Subject Alternative Name, CRLs - it's all there. Yet the Server Cert seems to be the one causing issues:

Begin validation of Certificate [Thumbprint 31F4A2CE105E232D1CD6F5FE3B66D837C361A2EA] issued to 'PrimarySite.company.domain.de'
 Completed validation of Certificate [Thumbprint 31F4A2CE105E232D1CD6F5FE3B66D837C361A2EA] issued to 'PrimarySite.company.domain.de'
 [CCMHTTP] ERROR: URL=https://PrimarySite.company.domain.de/SMS_MP/.sms_aut?SMSTRC, Port=443, Options=63, Code=0, Text=CCM_E_BAD_HTTP_STATUS_CODE
 [CCMHTTP] ERROR INFO: StatusCode=403 StatusText=Forbidden

As suggested, I checked CCMMessaging.log on one of the clients:

Raising event:
instance of CCM_CcmHttp_Status
{
 ClientID = "GUID:f07168f8-fb4d-4758-825b-c37419c26825";
 DateTime = "20220131135140.999000+000";
 HostName = "PrimarySite.company.domain.de";
 HRESULT = "0x00000000";
 ProcessID = 8480;
 StatusCode = 0;
 ThreadID = 9924;
};
OutgoingMessage(Queue='mp_[http]mp_locationmanager', ID={1C5395D9-744E-4957-A08B-C70F5CC683A4}): Delivered successfully to host 'PrimarySite.company.domain.de'.
Begin validation of Certificate [Thumbprint 31F4A2CE105E232D1CD6F5FE3B66D837C361A2EA] issued to 'PrimarySite.company.domain.de'
Completed validation of Certificate [Thumbprint 31F4A2CE105E232D1CD6F5FE3B66D837C361A2EA] issued to 'PrimarySite.company.domain.de'
Raising event:
instance of CCM_CcmHttp_Status
{
 ClientID = "GUID:f07168f8-fb4d-4758-825b-c37419c26825";
 DateTime = "20220131135141.128000+000";
 HostName = "PrimarySite.company.domain.de";
 HRESULT = "0x00000000";
 ProcessID = 8480;
 StatusCode = 0;
 ThreadID = 7564;
};
OutgoingMessage(Queue='mp_[http]mp_locationmanager', ID={8545DF89-9A4F-4520-942C-199D0640E3AA}): Delivered successfully to host 'PrimarySite.company.domain.de'.
Supplied sender token is null. Using GetUserTokenFromSid to find sender's token.
AAD Auth is not ready for user 'S-1-5-21-1024489538-160500420-XXXXXXXXX-7793'
Client doesn't have PKI issued cert and cannot get CCM access token. Error 0x8000ffff
[CCMHTTP] ERROR: URL=https://PrimarySite.company.domain.de/ccm_system_windowsauth/request, Port=443, Options=31, Code=0, Text=CCM_E_NO_TOKEN_AUTH
[CCMHTTP] ERROR INFO: StatusCode=403 StatusText=Forbidden
Raising event:
instance of CCM_CcmHttp_Status
{
 ClientID = "GUID:f07168f8-fb4d-4758-825b-c37419c26825";
 DateTime = "20220131135150.385000+000";
 HostName = "PrimarySite.company.domain.de";
 HRESULT = "0x87d00455";
 ProcessID = 8480;
 StatusCode = 403;
 ThreadID = 4504;
};
Successfully queued event on HTTP/HTTPS failure for server 'PrimarySite.company.domain.de'.
Post to https://PrimarySite.company.domain.de/ccm_system_windowsauth/request failed with 0x87d00231.

I'm aware that the log states the device doesn't have a PKI issued client cert, but that's not true. The cert is there, it's even used at some point and then just reverts to this error again. Of course I checked the SSL stores for the PKIs Root and Intermediate certs and they're both there in their respective stores, so no worries there.

Also checked if this might be a false positive, but once devices switch over to PKI, they no longer receive policies, so apps deployed to the specific test device don't show up in Software Center and existing Apps that had previously been deployed but not installed fail to install when clicked.

Totally stumped at the moment.

Cheers,

Fred

Rita Hu -MSFT 9,661 Reputation points

2022-02-21T09:35:37.833+00:00

Please help to review the ClientIDManagerStartup.log on the CM client to check whether the client registration successful. The log's location is C:\Windows\CCM\Logs.

In additioni, we could review the MP_RegistrationManager.log on the Site Server. The log records activities related to client registration, such as validating certificates, CRL, and tokens.

Regards,
Rita

Share via

CM HTTPS 403 Errors

4 answers

Your answer