ConfigMgr MP not responding

Hollisorama 381 Reputation points
2022-11-23T22:12:57.807+00:00

On 11/18/22, I started noticing the following MP error on our primary site server.

MP Control Manager detected management point is not responding to HTTP requests. The HTTP status code and text is 403, Forbidden.

These are the suggested action which I checked and all of them are configured correctly

Possible cause: Management point encountered an error when connecting to SQL Server.
Solution: Verify that the SQL Server is properly configured to allow Management Point access. Verify that management point computer account or the Management Point Database Connection Account is a member of Management Point Role (smsdbrole_MP) in the SQL Server database.
Possible cause: The SQL Server Service Principal Names (SPNs) are not registered correctly in Active Directory
Solution: Ensure SQL Server SPNs are correctly registered. Review Q829868.
Possible cause: Internet Information Services (IIS) isn't configured to listen on the ports over which the site is configured to communicate.
Solution: Verify that the designated Web Site is configured to use the same ports which the site is configured to use.
Possible cause: The designated Web Site is disabled in IIS.
Solution: Verify that the designated Web Site is enabled, and functioning properly.
Possible cause: The MP ISAPI Application Identity does not have the requisite logon privileges.
Solution: Verify that the account that the MP ISAPI is configured to run under has not been denied batch logon rights through group policy.
For more information, refer to Microsoft Knowledge Base article 838891.*

This MP happens to also be our site server. All of the other site server roles are functioning correctly. When I look at the CcmMessaging.log on this machine, I see the following log entries

EndpointMessage(Queue='MP_RelayEndpoint', ID={501A05FD-A71D-4380-BD0D-982010A1CD8D}): Will be discarded (0x8009200c). CcmMessaging 11/23/2022 4:03:22 PM 11496 (0x2CE8)
EndpointMessage(Queue='MP_RelayEndpoint', ID={0D5353B3-13B0-457A-B81D-63ED0E89F338}): Will be discarded (0x8009200c). CcmMessaging 11/23/2022 4:05:01 PM 11616 (0x2D60)
Supplied sender token is null. Using GetUserTokenFromSid to find sender's token. CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
AAD Auth is not ready for user 'S-1-5-21-709937114-2191035849-1797255849-13173' CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
Client doesn't have PKI issued cert and cannot get CCM access token. Error 0x8000ffff CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
[CCMHTTP] ERROR: URL=https://OSB-SCCM01.domain.local/ccm_system_windowsauth/request, Port=443, Options=63, Code=0, Text=CCM_E_NO_TOKEN_AUTH CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
[CCMHTTP] ERROR INFO: StatusCode=403 StatusText=Forbidden CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
Raising event:
instance of CCM_CcmHttp_Status
{
ClientID = "GUID:E6B1F88B-DEF2-4608-98A1-3A854172E9FB";
DateTime = "20221123220543.768000+000";
HostName = "OSB-SCCM01.domain.local";
HRESULT = "0x87d00455";
ProcessID = 7176;
StatusCode = 403;
ThreadID = 11496;
};
CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
Successfully queued event on HTTP/HTTPS failure for server 'OSB-SCCM01.domain.local'. CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)
Post to https://OSB-SCCM01.domain.local/ccm_system_windowsauth/request failed with 0x87d00231. CcmMessaging 11/23/2022 4:05:43 PM 11496 (0x2CE8)

I re-issued the client cert just to be sure and that does not resolve the issue.

Any thoughts on how to solve this are appreciated. I do have another server that acts as an MP so in the meantime, I have removed the MP role from the site server so that clients will be able to communicate properly.

Microsoft Configuration Manager
0 comments

6 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. CherryZhang-MSFT 6,481 Reputation points
    2022-11-26T12:17:28.267+00:00

    Hi @Hollisorama ,

    Thanks for your feedback and sharing. We're glad that the question is fixed now. Here's a short summary for the problem, we believe this will help other users to search for useful information more quickly. It's appreciated that you could click "Accept Answer" to the reply. If there's anything else we can help in the future, feel free to post in our Q&A to discuss together.

    Problem/Symptom:
    ConfigMgr MP not responding

    Reason:
    Replace a Trusted Publisher certificate for Ivanti Patch for SCCM and that product said needed to place that newly issued certificate in the Trusted Root Certificate store.

    Solution/Workaround:
    1, According to the record Verification of Certificate chain returned 800B0109 SMS_MP_CONTROL_MANAGER in mpcontrol.log. the two reference articles were found.
    https://social.technet.microsoft.com/forums/en-US/79d99027-0a40-4f00-8336-8c27b9e54fc4/management-point-error-call-to-httpsendrequestsync-failed-for-port-443-with-status-code-403-text
    https://learn.microsoft.com/en-US/troubleshoot/developer/webapps/iis/www-authentication-authorization/errors-403-7-reject-client-certificate-rquest
    2, According to the article, removed the new certificate from the Trusted Root Certificate store and the MP started responding correctly.

    Thanks again for your time and patience!

    Best regards,
    Cherry

    1 person found this answer helpful.
    0 comments
  2. CherryZhang-MSFT 6,481 Reputation points
    2022-11-24T09:14:26.51+00:00

    Hi @Hollisorama ,

    To narrow down the problem, we need more information.

    > I re-issued the client cert just to be sure and that does not resolve the issue.

    1, What certificate are you depicting? According to your description, you already configure Settings for Client PKI certificates, right? Can client and management point communicate normally before that? (If I misunderstand something, please let me know.)

    2, Which communication method are you using? HTTPS only or HTTPS or HTTP? Have you import any relevant PKI certificates? Could you please upload a screenshot for our reference?
    263852-picture3.png

    3, Please help check mpcontrol.log and LocationServices.log. If there are any useful information.
    mpcontrol.log: Navigate to (Site system server)…\Microsoft Configuration Manager\Logs\mpcontrol.log
    LocationServices.log: Navigate to (Client):C:\Windows\CCM\Logs\LocationsServices.log

    4, Can client Ping OSB-SCCM01.domain.local?

    5, For bindings in IIS, have you done any configuration?
    263853-picture4.png

    Looking forward to your reply!

    Best regards,
    Cherry

    0 comments
  3. Hollisorama 381 Reputation points
    2022-11-24T18:52:03.473+00:00

    Hi Cherry

    This site has been in operation for 9+ years and healthy. These client logs are from the same MP showing the error. I'm just showing both sides of the communication issue. The first log entry are errors from the mpcontrol.log. The second log entries are from the client running on that same MP. We use Client PKI and enforce HTTPS communications in the site. IIS bindings have the correct certificate and ping for the MP are functioning normally.

  4. Hollisorama 381 Reputation points
    2022-11-25T20:42:16.067+00:00

    Looking in the mpcontrol.log on the management point, I see these message.

    *Applied D:P(A;CIOI;GA;;;SY)(A;CIOI;GA;;;BA)(A;CIOI;GR;;;LS)(A;CIOI;GR;;;S-1-5-17) to folder E:\Program Files\Microsoft Configuration Manager\Client SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    SSL is enabled. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Client authentication is also enabled. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    CRL Checking is also enabled. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Machine name is 'OSB-SCCM01.domain.local'. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Begin validation of Certificate [Thumbprint b8c48ec0f3d199981aa964832b2831157353538a] issued to 'OSB-SCCM01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Certificate doesn't have "SSL Client Authentication" capabilities. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Completed validation of Certificate [Thumbprint b8c48ec0f3d199981aa964832b2831157353538a] issued to 'OSB-SCCM01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Skipping this certificate which is not valid for ConfigMgr usage. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Begin validation of Certificate [Thumbprint 2f8d94cf7a41387a593df3590b5b7bdae460f353] issued to 'OSB-SCCM01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Certificate has "SSL Client Authentication" capability. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Completed validation of Certificate [Thumbprint 2f8d94cf7a41387a593df3590b5b7bdae460f353] issued to 'OSB-SCCM01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)

    >> Selected Certificate [Thumbprint 2f8d94cf7a41387a593df3590b5b7bdae460f353] issued to 'OSB-SCCM01.domain.local' for HTTPS Client Authentication SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)

    Call to HttpSendRequestSync failed for port 443 with status code 403, text: Forbidden SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Inbox source is local on OSB-SCCM01.domain.LOCAL SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Sent summary record of SMS Management Point on ["Display=\OSB-SCCM01.domain.LOCAL\"]MSWNET:["SMS_SITE=ONB"]\OSB-SCCM01.domain.LOCAL\ to \OSB-SCCM01.domain.local\SMS_ONB\inboxes\sitestat.box\tgl8t6e2.SUM, Availability 1, 366998524 KB total disk space , 101174792 KB free disk space, installation state 0. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Http test request failed, status code is 403, 'Forbidden'. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Successfully performed Management Point availability check against local computer. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    SSL is enabled. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    CRL Checking is also enabled. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Using thread token for request SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Call to HttpSendRequestSync succeeded for port 443 with status code 200, text: OK SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Http test request succeeded. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    STATMSG: ID=5465 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_MP_CONTROL_MANAGER" SYS=OSB-SCCM01.domain.LOCAL SITE=ONB PID=13304 TID=12456 GMTDATE=Fri Nov 25 07:48:22.334 2022 ISTR0="" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0 LE=0X0 SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Successfully performed User Service availability check against local computer for /CMUserService_WindowsAuth/applicationviewservice.asmx. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Beginning periodic tasks. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    RetryWINSOperationIfNecessary: No need to retry. Returning. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    RetryDNSPublishingIfNecessary: No need to retry. Returning. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Found SSL binding '64199E03746C76DF3D7576D771816723AC4A4BE9', 'My' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Begin searching client certificates based on Certificate Issuers SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Completed searching client certificates based on Certificate Issuers SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Begin to select client certificate SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Using certificate selection criteria 'CertHashCode:64199E03746C76DF3D7576D771816723AC4A4BE9'. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Begin validation of Certificate [Thumbprint 64199E03746C76DF3D7576D771816723AC4A4BE9] issued to 'osb-sccm01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Completed validation of Certificate [Thumbprint 64199E03746C76DF3D7576D771816723AC4A4BE9] issued to 'osb-sccm01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)

    >> Client selected the PKI Certificate [Thumbprint 64199E03746C76DF3D7576D771816723AC4A4BE9] issued to 'osb-sccm01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)

    Begin validation of Certificate [Thumbprint 64199E03746C76DF3D7576D771816723AC4A4BE9] issued to 'osb-sccm01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Verification of Certificate chain returned 800B0109 SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Completed validation of Certificate [Thumbprint 64199E03746C76DF3D7576D771816723AC4A4BE9] issued to 'osb-sccm01.domain.local' SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Certificate is not issued by SCCM. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    SSL binding on port 443 isn't with CCM genreated cert. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Started User Service maintenance... SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Certificate is Exportable SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Successfully granted permission to certificate SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    User Service: retrieved certificate fc 01 b1 cd ac e3 22 f1 a5 bb 65 10 c7 78 b1 90 d2 ce 2b 19 SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Successfully completed User Service maintenance. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)
    Completed periodic tasks. SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)*

    0 comments
  5. Hollisorama 381 Reputation points
    2022-11-26T06:12:04.63+00:00

    I was able to solve the issue from https://social.technet.microsoft.com/forums/en-US/79d99027-0a40-4f00-8336-8c27b9e54fc4/management-point-error-call-to-httpsendrequestsync-failed-for-port-443-with-status-code-403-text and https://learn.microsoft.com/en-US/troubleshoot/developer/webapps/iis/www-authentication-authorization/errors-403-7-reject-client-certificate-rquest

    Last week we had replaced a Trusted Publisher certificate for Ivanti Patch for SCCM and that product said we needed to place that newly issued certificate in the Trusted Root Certificate store. That is when this issue started.

    I removed that new certificate from the Trusted Root Certificate store and the MP started responding correctly.

    The key line from the log entry above is Verification of Certificate chain returned 800B0109 SMS_MP_CONTROL_MANAGER 11/25/2022 1:48:22 AM 12456 (0x30A8)

    That error code number allowed me to find the reference articles above.

    0 comments