CMG - Azure Virtual Machine Scale Set Cloud Management Gateway instance is Unhealthy in Azure

Question

CMG - Azure Virtual Machine Scale Set Cloud Management Gateway instance is Unhealthy in Azure

Flint 26

Hi All,

In our SCCM environment configured with Cloud Management Gateway. Its stopped working from last One month.

In SCCM console CMG connection point show as disconnected.

We checked in Azure Virtual Machine Scale Set Cloud Management Gateway instance is unhealthy.

We have deleted the unhealthy instance, but after creating the new instance also show as Unhealthy state.

The port 10124 and 10125 Test-NetConnection is success on SCCM Primary Server.

Anonymous

2024-10-03T08:47:07.24+00:00
Hi Flint,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

It sounds like your customer is experiencing issues with their Cloud Management Gateway (CMG) in Azure. Here are some steps and considerations to help troubleshoot and resolve the issue:

Make sure that the CMG connection point is correctly configured and connected to the primary site. Check that there is no network security groups or firewalls blocking the connection.

Examine the logs for both the CMG connection point and the CMG itself. Look for any errors or warnings that might indicate the cause of the unhealthy state.

Verify the health of the Azure VM Scale Set. Ensure all instances are running correctly and there are no issues with the underlying Azure infrastructure.

Confirm that the certificates used by the CMG are valid and correctly configured. Expired or misconfigured certificates can lead to connectivity issues.

Since the Test-NetConnection for ports 10124 and 10125 is successful, it indicates that the primary server can reach the CMG. Ensure all required ports and protocols are open and not blocked by any network security groups or firewalls.

If the issue persists, consider deleting and recreating the CMG. Follow all steps correctly during the setup process to ensure proper configuration.

For more detailed guidance, please refer to the documentation on setting up and troubleshooting CMG:

Set up CMG - Configuration Manager

Plan for CMG - Configuration Manager

CMG performance and scale

If you have any further queries, do let us know. If the comment is helpful, please click "Upvote".
Flint 26 Reputation points

2024-10-03T13:21:52.31+00:00
Hi Sai,

Its previously working fine without any issue.

I have checked the log files for certification expiry from Client Machine( C:\Windows\CCM\Logs\CcmMessaging.log), but it's looks good.

We have configured the 2 VM Instances in CMG. After deleting the unhealthy VM instance in Azure VM Scale Set, it will create again the Unhealthy VM instance.

Analyze the Cloudmgr.log log file, but not able to find out any error.

We analyze the smsAdminUI.log, get the below error in Port 443, but firewall is disabled on the SCCM Primary Server where CMG configured. Check with Network team they also confirmed no blockage in 443 port.

smsAdminUI.log

[13, PID:28384][10/01/2024 18:33:01] :System.Net.WebException\r\nUnable to connect to the remote server\r\n at System.Net.HttpWebRequest.GetResponse()

at Microsoft.ConfigurationManagement.AdminConsole.AzureServices.CMGAnalyzer.backgroundWorker_DoWork(Object sender, DoWorkEventArgs e)\r\nSystem.Net.Sockets.SocketException\r\nA connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xxx.xxx.xxx:443\r\n at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)\r\n
Anonymous

2024-10-04T04:07:17.04+00:00

Hi Flint,

Thank you for the additional information. Since the CMG was previously functioning without issues and you have confirmed that the certificates are valid and there are no port 443 blockages, here are further steps to troubleshoot:

Verify the Health of the Azure VM Scale Set (VMSS):
Although you have deleted the unhealthy VM instance, it seems the new instance is still being marked as unhealthy. Please check the health of the entire VM Scale Set in Azure.

Go to the Azure Portal > Virtual Machine Scale Sets > Your CMG instance > Instances. Ensure all VM instances are running properly and there are no errors related to the infrastructure.

Azure Virtual Machine Scale Sets - Troubleshooting

Network Security and Firewall:
Although the firewall on your SCCM Primary Server is disabled, verify that there are no issues with the Network Security Groups (NSGs) in Azure. Ensure that NSGs allow port 443 for inbound and outbound traffic between the SCCM server and the CMG in Azure.

Network security groups (NSGs) in Azure
Azure firewall logs and troubleshooting

Azure Load Balancer Configuration:
If your CMG uses an Azure Load Balancer, ensure that the health probes are correctly configured to check the VM instances' status. Misconfigured health probes may cause the VMSS instances to report as unhealthy.

Azure Load Balancer Troubleshooting

CMG Analyzer Tool:
I recommend using the CMG Analyzer tool in the SCCM console to check the connectivity between the CMG, SCCM, and Azure. This will help identify any underlying connection issues. The error you mentioned in smsAdminUI.log may indicate a transient network or infrastructure issue.

Check DNS and Resource Limits:
Ensure that DNS resolution between the SCCM server and the CMG is working correctly. Also, check your Azure subscription quotas and limits to confirm that resource constraints are not causing the issue.

For additional information, please refer to the following documentation: Monitor the Cloud Management Gateway - Configuration Manager

If you have any further queries, do let us know. If the comment is helpful, please click "Upvote".
Anonymous

2024-10-07T05:00:24.3266667+00:00

Hi Flint,

Just checking in to see if you had a chance to review my comment on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.
Clifton Hughes 0 Reputation points

2025-01-24T23:59:54.6733333+00:00

I have the same issue in my lab, this was not a problem before 2409 for me, however, new cert, and no firewalls to block anything, and I am unable to connect to the CMG Service even after rebuilding it multiple times, and it is either getting a provisioning error that is not showing up in the Configmgr console, or it shows as running, but Unhealthy. Three tries now in different regions and with my Cname FQDN and a valid matching cert or the deployment name FQDN and matching cert. This is the provisioning error it throws on creation of the VM Scale set for the CMG. VM has reported a user failure when processing extension 'InstallCMG'. Please correct the error and try again. (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error code: '2'. Error message: 'Command execution finished, but failed because it returned a non-zero exit code of: '1''. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot.

1 answer

Your answer

Anonymous

2024-10-03T08:47:07.24+00:00

Hi Flint,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

It sounds like your customer is experiencing issues with their Cloud Management Gateway (CMG) in Azure. Here are some steps and considerations to help troubleshoot and resolve the issue:

Make sure that the CMG connection point is correctly configured and connected to the primary site. Check that there is no network security groups or firewalls blocking the connection.

Examine the logs for both the CMG connection point and the CMG itself. Look for any errors or warnings that might indicate the cause of the unhealthy state.

Verify the health of the Azure VM Scale Set. Ensure all instances are running correctly and there are no issues with the underlying Azure infrastructure.

Confirm that the certificates used by the CMG are valid and correctly configured. Expired or misconfigured certificates can lead to connectivity issues.

Since the Test-NetConnection for ports 10124 and 10125 is successful, it indicates that the primary server can reach the CMG. Ensure all required ports and protocols are open and not blocked by any network security groups or firewalls.

If the issue persists, consider deleting and recreating the CMG. Follow all steps correctly during the setup process to ensure proper configuration.

For more detailed guidance, please refer to the documentation on setting up and troubleshooting CMG:

Set up CMG - Configuration Manager

Plan for CMG - Configuration Manager

CMG performance and scale

If you have any further queries, do let us know. If the comment is helpful, please click "Upvote".
Flint 26 Reputation points

2024-10-03T13:21:52.31+00:00

Hi Sai,

Its previously working fine without any issue.

I have checked the log files for certification expiry from Client Machine( C:\Windows\CCM\Logs\CcmMessaging.log), but it's looks good.

We have configured the 2 VM Instances in CMG. After deleting the unhealthy VM instance in Azure VM Scale Set, it will create again the Unhealthy VM instance.

Analyze the Cloudmgr.log log file, but not able to find out any error.

We analyze the smsAdminUI.log, get the below error in Port 443, but firewall is disabled on the SCCM Primary Server where CMG configured. Check with Network team they also confirmed no blockage in 443 port.

smsAdminUI.log

[13, PID:28384][10/01/2024 18:33:01] :System.Net.WebException\r\nUnable to connect to the remote server\r\n at System.Net.HttpWebRequest.GetResponse()

at Microsoft.ConfigurationManagement.AdminConsole.AzureServices.CMGAnalyzer.backgroundWorker_DoWork(Object sender, DoWorkEventArgs e)\r\nSystem.Net.Sockets.SocketException\r\nA connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xxx.xxx.xxx:443\r\n at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)\r\n
Anonymous

2024-10-07T05:00:24.3266667+00:00

Hi Flint,

Just checking in to see if you had a chance to review my comment on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.
Clifton Hughes 0 Reputation points

2025-01-24T23:59:54.6733333+00:00

I have the same issue in my lab, this was not a problem before 2409 for me, however, new cert, and no firewalls to block anything, and I am unable to connect to the CMG Service even after rebuilding it multiple times, and it is either getting a provisioning error that is not showing up in the Configmgr console, or it shows as running, but Unhealthy. Three tries now in different regions and with my Cname FQDN and a valid matching cert or the deployment name FQDN and matching cert. This is the provisioning error it throws on creation of the VM Scale set for the CMG. VM has reported a user failure when processing extension 'InstallCMG'. Please correct the error and try again. (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error code: '2'. Error message: 'Command execution finished, but failed because it returned a non-zero exit code of: '1''. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot.

Answer 1

XinGuo-MSFT 22,271

Hi,

It sounds like you’re dealing with a tricky issue.

Here are a few steps you can take to troubleshoot and resolve the problem:

Please check the configuration of your CMG in the SCCM console. Ensure that all settings, including certificates and VM instance counts, are correctly configured.
Look at the logs and diagnostics for both the CMG and the Azure VM Scale Set. This can provide more insights into why the instances are marked as unhealthy.

Flint 26 Reputation points

2024-10-03T10:43:45.7133333+00:00
Hi,

I have checked the log files for certification expiry from Client Machine( C:\Windows\CCM\Logs\CcmMessaging.log), but it's looks good.

We have configured the 2 VM Instances in CMG. After deleting the unhealthy VM instance in Azure VM Scale Set, it will create again the Unhealthy VM instance.

Analyze the Cloudmgr.log log file, but not able to find out any error.

We analyze the smsAdminUI.log, get the below error in Port 443, but firewall is disabled on the SCCM Primary Server where CMG configured. Check with Network team they also confirmed no blockage in 443 port.

smsAdminUI.log

[13, PID:28384][10/01/2024 18:33:01] :System.Net.WebException\r\nUnable to connect to the remote server\r\n at System.Net.HttpWebRequest.GetResponse()

at Microsoft.ConfigurationManagement.AdminConsole.AzureServices.CMGAnalyzer.backgroundWorker_DoWork(Object sender, DoWorkEventArgs e)\r\nSystem.Net.Sockets.SocketException\r\nA connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xxx.xxx.xxx:443\r\n at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)\r\n
XinGuo-MSFT 22,271 Reputation points

2024-10-04T07:53:09.82+00:00

Since the error in smsAdminUI.log mentions an issue with port 443, double-check the SSL/TLS configuration on your CMG and ensure that the certificates are correctly installed and not expired. Even though the firewall is disabled, there might be issues with SSL/TLS handshakes.

Double-check the inbound and outbound rules for your NSGs and any Azure Firewall configurations to confirm that port 443 is open and not being blocked. This might help resolve the connectivity issue you’re experiencing.