Skype for Business Hybrid Voice Team https://docs.microsoft.com/archive/blogs/sfbhybridvoice/feed.xml Blog of Skype for Business Hybrid Voice team Wed, 17 Apr 2019 03:47:54 GMT en-US hourly 1 How to Update Cloud Connector to Version 1.4.2 https://docs.microsoft.com/archive/blogs/sfbhybridvoice/how-to-update-cloud-connector-to-version-1-4-2 Mon, 08 May 2017 14:14:01 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=255 Https://aka.ms/CloudConnector. For troubleshooting upgrade issues, please see Troubleshooting Cloud Connector Edition 1.4.2 Updates and New Deployments

Prepare for Upgrade

Schedule a maintenance window for the work. If you have a single appliance and are updating from a build prior to 1.4.1 (version 1.3.X), you will need to plan for downtime in PSTN calling during the upgrade process which will take between 1.5 and 2 hours. If you have CCE version 1.4.1 the downtime in PSTN calling will only be a few minutes when switching from the previous version to the latest version. Gather credentials that you will need for the deployment used when running Register-CcAppliance. You will need local Cloud Connector administrator password and Office 365 Tenant administrative password. Backup your existing Cloud Connector configuration file for reference (see below for details) when creating a new configuration file for the latest version. After installing latest Skype for Business Online Cloud Connector, you will obtain the latest version file by running Export-CcConfigurationSampleFile from administrative PowerShell.

Update from 1.3.4 to 1.4.2

As there were significant changes since 1.3.4, you need to clean up the 1.3.4 deployment before installing the latest version.

Remove 1.3.4

  1. Backup the %userprofile%\CloudConnector\CloudConnector.ini to another drive so it will be available for reference when creating the new CloudConnector.ini file.
  2. Stop all the Cloud Connector VM’s in Hyper-V Manager on the Cloud Connector host appliance.
  3. Delete the Cloud Connector VM’s and virtual disks which are in the <Appliance Directory>\Instances folder. Run Get-CcApplianceDirectory cmdlet from administrative PowerShell to find the directory if not known.
  4. If you want to use the existing external Edge certificate, back it up. Run Get-CcExternalCertificateFilePath to find the current location of the certificate.
  5. Delete all content from the following folders
    1. %userprofile%\Cloudconnector.
    2. Appliance directory. Run Get-CcApplianceDirectory [BH3] to find the location of the directory.
    3. Site directory. Run Get-CcSiteDirectory to find the location of the directory.

Prepare the Appliance

Gather credentials that you will need for the deployment used when running Register-CcAppliance. You will need local Cloud Connector administrator password and Office 365 Tenant administrative password. Follow all instructions in Prepare your Cloud Connector appliance including steps to create a new base VHDX for the Cloud Connector virtual machines. Note – be sure that this step completes successfully. If the Base VM needs special network configuration, e.g. Proxy settings, to connect to the Internet for obtaining windows updates, be sure to use the PauseBeforeUpdate setting to allow you to connect to the VM and update network settings to allow connectivity before continuing. When updating the Cloud Connector configuration file, refer to the backup copy you made in step 3 above. Also, if you manually created a Hybrid PSTN site in your tenant, be sure that the Site name and Edge FQDN defined in the Cloud Connector configuration file matches site settings in your tenant. Run Get-CsHybridPSTNSite in Skype for Business Online PowerShell to see sites defined in your tenant. In the latest version, Register-CcAppliance cmdlet will either create a new site based on name defined in configuration, or update existing site if a site with same name exists in your tenant.

Install the latest version

  1. Follow instructions to Deploy the first Skype for Business Cloud Connector Edition appliance in Deploy a single site in Cloud Connector.
  2. Confirm Appliance status in Skype for Business Online Admin Center > Voice > on premises PSTN, or by running Get-CsHybridPSTNAppliance using Skype for Business Online PowerShell.
  3. Validate the new deployment following Validate your Cloud Connector deployment.

Update from 1.3.8 to 1.4.2

  1. Backup the <Appliance Root>\CloudConnector.ini for reference when updating CloudConnector.ini file for the latest version. Run Get-CcApplianceDirectory to find location.
  2. Uninstall the current appliance. To do this, start a PowerShell console as administrator and run Uninstall-CcAppliance.
  3. Follow instructions to Deploy the first Skype for Business Cloud Connector Edition appliance in Deploy a single site in Cloud Connector.
  4. Confirm Appliance status in Skype for Business Online Admin Center > Voice > on premises PSTN, or by running Get-CsHybridPSTNAppliance using Skype for Business Online PowerShell.
  5. Validate the new deployment following Validate your Cloud Connector deployment.

Update from 1.4.1 to 1.4.2

  1. Remove the Forward PAI section in the <Appliance Root>\CloudConnector.ini. Run Get-CcApplianceDirectory to find location. Forward PAI has been deprecated in 1.4.2 as it’s no longer required to resolve missing caller ID for outbound calls from Skype for Business Mobile clients which has been resolved with changes in outbound routing logic for Cloud Connector.
  2. Follow instructions to Upgrade a single site to a new version Upgrade to a new version of Cloud Connector.
  3. Confirm Appliance status in Skype for Business Online Admin Center > Voice > on premises PSTN, or by running Get-CsHybridPSTNAppliance using Skype for Business Online PowerShell.
  4. Validate the new deployment following Validate your Cloud Connector deployment.
]]> Troubleshooting Cloud Connector Edition 1.4.2 Updates and New Deployments https://docs.microsoft.com/archive/blogs/sfbhybridvoice/troubleshooting-cloud-connector-edition-1-4-2-updates-and-new-deployments Fri, 07 Apr 2017 14:19:11 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=245 https://aka.ms/cloudconnetor.

Cloud Connector Download Failed

Cloud Connector Software is download failing with: Failed to download installation files with exception: Exception calling "DownloadFile" with "2" argument(s): "Unable to connect to the remote server".

Troubleshooting

Confirm that one Host Appliance physical adapter is configured with a gateway that provides a route to the Internet
  1. On Host Appliance, open a command prompt and run: Ipconfig /All
Confirm that a gateway is set on one of the network adapters. Confirm that the Host Appliance server has Internet connectivity and DNS resolution of external resources
  1. Open command prompt on the host and run and confirm reply: Ping Bing.com
If you use a Proxy in the DMZ for Internet access, confirm that the Proxy settings are per-machine rather than by user. Otherwise Cloud Connector downloads will fail. Set either with registry change or Group Policy setting:
  1. Registry: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings] ProxySettingsPerUser dword:00000000
  2. Group Policy: Computer>Administrative Templates>Windows Components> Internet Explorer: Make Proxy settings per-machine (rather than per user
Confirm firewall rules allow traffic to the following:
  1. TCP Port 443: https://go.microsoft.com
  2. TCP Port 80: http://download.microsoft.com

Install of 1.4.2 Fails with Access Denied Connecting to Remote Server

The following error displayed during update to version 1.4.2 of the Cloud Connector appliance: Connecting to remote server 192.168.213.2 failed with the following error message: Access is denied. This is because the Domain Admin and VM Admin credentials for the Cloud Connector servers expired. To resolve:
  1. Log in to the CCE AD machine with the CCE Domain Admin password. Open the AD Users and Computers console from Administrative tools, and reset both the Administrator and CCEService account passwords to the original password and set the passwords as non-expiring.
  2. Log in to each of the other CCE machines with the CCE Domain Admin password. Open Computer Management, and reset both the Administrator and CCEService accounts to the original passwords and set to non-expiring passwords.
Note that in 1.4.2 and later all Cloud Connector account credentials are non-expiring and this error would be unexpected unless someone changed credentials to be different that configured in Cloud Connector cached passwords set with Register-CcAppliance.

Install of 1.4.2 Fails with WinRM Error Connecting to Remote Server

The following error displayed during installation of the new Cloud Connector appliance: Connecting to remote server 192.168.213.2 failed with the following error message: WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. If you use a Proxy in the DMZ for Internet access, set WinHTTP Proxy settings on the Host appliance with your proxy server and a Bypass-list including the "192.168.213.*” network used by your Cloud Connector Managements services to bypass the proxy. Otherwise management connectivity will fail and prevent the deployment of Cloud Connector Edition. Note: Match the bypass list to the management subnet that defined in your CloudConnector.ini file. Sample winhttp configuration command: netsh winhttp set proxy "10.10.10.175:8080" bypass-list="*.local;1.*;172.20.*;192.168.218.*'<local>"

Missing CCE Management Service: Install Fails After Uninstalling Current Version of Skype for Business Cloud Connector Edition

The update to 1.4.2 fails after the CCE Management Service stops and the current version of Skype for Business Cloud Connector Edition uninstalls, and use see the following error displayed: Uninstallation failed with exception System.Management.Automation.MethodInvocationException: Exception calling "Uninstall" : "Illegal operation attempted on a registry key that has been marked for deletion. The Group Policy to prevent removal of registry on logoff was either not created or removed in the case of a domain joined Host appliance. To resolve:
  1. Make the Group Policy entry described in Prepare Cloud Connector Appliance https://technet.microsoft.com/en-us/library/mt740649.aspx
  2. Follow Steps 2-7 here if not in Bits Update time window and want to manually update, https://technet.microsoft.com/en-us/library/mt740656.aspx, or, steps 2-4 if you are.

Install Fails Connecting to Machine After Waiting 600 Seconds

Installation fails during deployment of the first Cloud Connector VM with: Can’t connect to machine 192.168.213.2 after waiting 600 seconds A network issue is preventing the Host appliance from making a PowerShell session to the Cloud Connector Virtual Machine over the management switch. Troubleshooting Confirm no VLAN’s configured on the Host appliance. If using a Proxy server, confirm WinHTTP proxy settings bypass list includes management switch network as described above. Check if the switches on the Cloud Connector VM have Automatic Private IP Addresses of 169.x. It they have 169.x addresses, the Convert-CcIsoToVhdx did not complete correctly. To resolve:
  1. Run: Convert-CcIsoToVhdx -IsoFilePath <Windows ISO File Path, including file name> -PauseBeforeUpdate
  2. When prompted, connect to the Base VM using credentials provided in the PowerShell output.
  3. Make any necessary adjustments to allow the computer to connect to Windows Update Service.
  4. Complete all Windows Updates and restart the VM.
  5. Return to PowerShell and answer prompt complete the process.
]]> Cloud Connector Custom Configurations https://docs.microsoft.com/archive/blogs/sfbhybridvoice/cloud-connector-custom-configurations Wed, 05 Apr 2017 14:44:41 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=235 Plan Cloud Connector Edition documentation. The INI file is the main source of configuration information for Cloud Connector.  During the update process, Cloud Connector installs the fresh set of Virtual machines and applies configuration settings from the INI file. Any parameters that are not present in INI are set to the default values. Tenant administrators might have a need to change other parameters, that are not part of the INI to provide interoperability with 3rd party gateways or SBCs or for billing purposes. In this case, administrators need to disable auto update, perform the manual update, and return the settings after every update. We are working on a better management story moving forward. For example, we receive many questions on how to enable History-info and Referred-By headers which are disabled by default. By enabling these headers enterprises can more effectively bill for their services for calls forwarded or transferred to another PSTN number. More about the headers are here: http://download.microsoft.com/download/2/4/5/245583A7-E9E1-403B-BD15-AB711C7DF744/Lync%202013%20Headers%20for%20SIP%20trunk%20provider%20billing.docx To set these headers, run the following in PowerShell on each Cloud Connector mediation server deployed: Set-CSTrunkConfiguration -ForwardCallHistory $true. However, when Cloud Connector is updated, the headers will be set to the default value of $false. For custom configurations, we recommend the following:
  1. Set auto update to Manual instead of Automated. To set manual update, run the following in Skype for Business Online Powershell for every Cloud Connector PSTN site:
Set-CsHybridPSTNSite -EnableAutoupdate $false.
  1. When an update is available, update one appliance in a site at a time as follows:
  2. Follow instructions here to manually update the appliance: https://technet.microsoft.com/EN-US/library/mt740656.aspx Step 7 (Switch-CcVersion). Important: do not perform Step 7 (Switch-CCVersion) yet. Switching the version will bring the appliance to the production before you reapplied the custom configuration
  3. Apply the custom changes, for example by running Set-CSTrunkConfiguration -ForwardCallHistory $true in PowerShell on the Cloud Connector mediation server
  4. Run Step 7 (Switch-CCVersion) from https://technet.microsoft.com/EN-US/library/mt740656.aspx
Note: Microsoft supports the previously released version of Cloud Connector for 60 days after a new version is released. Cloud Connector appliances must be updated within 60 days of a new release.    ]]> Cloud Connector Calls: Identifying call quality via Call Quality Dashboard https://docs.microsoft.com/archive/blogs/sfbhybridvoice/cloud-connector-calls-identifying-call-quality-via-call-quality-dashboard Mon, 27 Mar 2017 13:01:33 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=155 After the deployment of Cloud Connector (or any telephony option) the first question administrators usually ask is how good the call quality is? Microsoft has a nice tool Call Quality Dashboard which allows seeing and analyzing call quality. The detailed description of Call Quality Dashboard is here First a little bit of theory. When a client places a call there are two streams going on, the first one is RTP and the second is RTCP. RTP stands for Real Time Protocol and is used to transmit audio and video. RTCP is Real Time Control Protocol and is used for controlling the media flow and also used to collect network information during the call. Such statistics includes measuring network jitter, round-trip, packet loss. After a call client combines the data about devices, client versions, network ranges and many others, includes network statistics the client received from RTCP, and sends in a “Service” message to Skype for Business Online servers that processes the data and publishes it on tenant’s Call Quality Dashboard. Note there could be a delay up to  48 hours between client sent the data and seeing it on Call Quality Dashboard. The delay caused by internal processing jobs. You can find the data sent by the client in local client logs. Find a “Service” message after a call, and you will see what parameters the client sends. clinetlog Note the client does not send the logs if a call which was 5 sec or less. Now the data is in Call Quality Dashboard, the question is how to see the data about PSTN calls that were made using a Cloud Connector. If you look at CQD, there is information about all audio streams,  including peer to peer VoIP calls, audio, and video conferences. There is also a chance that you might have Microsoft PSTN calling for users in the US or UK while providing PSTN services using a hybrid voice option in countries where Microsoft PSTN Calling is not available. You can distinguish hybrid voice PSTN calls from any other calls you can use a parameter called “First User Agent” and filter by RTCC/6.0.0.0/* Back to the theory again. Parameters “First User Agent” and “Second User Agent” capture endpoints which were used in the particular call. On the screenshot below, an example of “First User Agent” endpoints from one of the tenants. End users utilized these devices when placing or receiving the call. The similar list you will see for the “Second User Agent”. fua   The very important point is that when an endpoint places a call to a service or receives call from a service (not a peer to peer call), the service endpoint is always reported in “First User Agent” regardless of a call flow and who call whom. The service which handles calls for Cloud Connector users is on-premises mediation server, which has version 6.0.0.0/* If you have calls that went through Microsoft PSTN calling the Mediation server version will be 7.0 Based on these two factors to distinguish Hybrid PSTN calls from all other audio streams you need only apply the filter: “First Usage Agent” = RTCC/6.0.0.0/* On screenshot is basic representation Dimensions, Measurements and Filters configuration to see Hybrid voice calls. cqd3   You can configure additional filters, for example, see network parameters during calls, or media connectivity issues to troubleshoot issues with call quality. You can find how to configure CQD to troubleshoot networks in Skype Operations Framework. In this blog, we want to show how to distinguish Cloud Connector calls from other calls. The result shown on the screenshot below. cqd4  
Questions: Question: I see many RTCC/6.0.0.0/LyncSkypeGateway/* as a “First User Agent” is it related to hybrid voice? Answer: No, these are gateways the service uses to place calls between Skype and Skype for Business users
Question: In some cases, I see “RTCC/6.0.0.0/* as a “Second User Agent” as on screenshot below. Based on the fact that if a call goes to the service, I expect RTCC/6.0.0.0 only as a “First User Agent.” Why?
pic5
Answer: These are forwarded calls. Imagine the situation where an end user received a call on a Lync Client but set forward to another number; the call goes back to mediation server to make an external call. Question: Why in some cases I see no data about call quality or network parameters? Answer: There are several cases when this data is not available: -          Calls to voice mail don’t send the data; -          About 1% of sending the data might not reach the processing service. This is service limitation and we allow some data not reaching the service; -          Forwarded calls also do not have the call quality data (will be fixed soon)
 ]]> Skype for Business Cloud Connector Version 1.4.2 Release https://docs.microsoft.com/archive/blogs/sfbhybridvoice/skype-for-business-cloud-connector-version-1-4-2-release Mon, 20 Mar 2017 07:12:10 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=125 Skype for Business Cloud Connector Version 1.4.2 Release This is an important update as it is the first update that will automatically update all existing installed Skype for Business Cloud Connector (Cloud Connector) 1.4.1 appliances based on the update schedule that administrators have configured for their Cloud Connector Hybrid PSTN Sites. For details please refer to Understanding the automatic update process for Cloud Connector Edition. Be sure to read Prepare for Cloud Connector Edition Release 1.4.2 prior to your update schedule to insure a successful update. If you are running version 1.3.8, please refer to the manual update instructions Upgrade a single site to a new version in the Cloud Connector Edition configuration guide. Also, be sure to export a new sample Cloud Connector configuration file after 1.4.2 has been installed and update your existing configuration file with new parameters, e.g., HardwareType and WSUS Server.

Thoughts and Motivations behind 1.4.2

As you’ll see in the sections below we have several fixes and improvements in this latest version of CCE. We’ve listened to your feedback and made the 1.4.X CCE easier to deploy, more robust and fixed many minor bugs. There are several reasons why we decided to issue a release now:
  • 1.4.1 was about 9 months old. There were numerous improvements to the components of CCE and there were a pile of improvements and workarounds that were starting to cause deployment friction with our customers.
  • 1.4.1 introduced Auto Update (among other improvements) but we had not released an actual update to verify at scale. While we tested Auto Update as part of the 1.4.1 release process, it was important to have a real update work well across the wide diversity of live CCE deployments. V2.0 is coming later this year and it has some substantial feature updates and changes to the CCE architecture, so it was important to make sure the fundamentals of 1.4.X Auto Update were well in hand before we start automatically updating a big release like V2.0
There is another important point to make here: It is really important to enable Auto Update. Unlike pure cloud offerings or pure on-premises offerings (like Skype for Business Server or Cisco Call Manager) the software needed to deliver CloudPBX and Hybrid PSTN separate into two distinct places. The other offerings have all their elements in one place and while there may be new releases or updates, the old releases are self-consistent and unchanging. If you can live with current features, you can delay updating. It’s not unusual for on-premises software to skip several updates. For Hybrid PSTN, the software components split across the Skype for Business Online (SFBO) cloud service elements and the elements inside the CCE in your data center. In SFBO, all components are capable of updating globally every 2 weeks and some can be done as quickly as 2 days. Some changes in the cloud need an upgrade to the on-premises CCE. If CCE does not auto update there is a real chance that PSTN calling will cease to work, or work intermittently. Deferring or stopping all CCE updates does not stop the cloud parts of your solution from updating. Auto Update is needed to ensure your CCE stays in sync with the Skype for Business cloud.

Changes in Support Policy

Since the beginning the EULA required before downloading CCE software notified the customer that their SLA requires that they comply with recommended software updates to keep their financially backed SLA in force. In support of that the CCE team spent a lot of time developing and perfecting the Auto Update subsystem for CCE to make staying current painless and eliminate software mismatch induced errors. Starting with CCE 1.4.2 we will be a little more prescriptive on updates and SLAs: When we release version N (1.4.2 in this case) there is a 60 day window within which version N-1 (the prior released version, 1.4.1) will also be supported against the SLA commitment. After 60 days only version N is supported against the SLA commitment. Many customers still are running v1.3.8. This version is the original version of CCE, and it will not automatically update but it will manually update to the current production version of CCE. Given how old this release is to the evolving CloudPBX software, the risk of failure or degradation grows day by day. Please update your 1.3.8 ASAP. No one wants users to lose the ability to make/receive phone calls. We have tested updating 1.3.8 to 1.4.1 and to 1.4.2. If running 1.3.8, please manually update to 1.4.2 now.

Improvements in 1.4.2

Revised cmdlet

Added following two parameters to Set-CcExternalCertificateFilePath: Target: Define the certificate target, either EdgeServer or MediationServer.
  • Set-CcExternalCertificateFilePath -Path <Path to Edge PFX Cerfiticate> -Target EdgeServer Set the path to the certificate that has private key which the script will import and assign to the external interface of the Edge server during deployment.
  • Set-CcExternalCertificateFilePath -Path <Path to PSTN Gateway Certificate> -Target MediationServer. Set the path to the certificate that has the certificate chain of the issuing CA for the PSTN Gateway which the script will import to the Mediation server certificate store during deployment.
Import: Import the certificate for the Edge server or Mediation server.
  • Set-CcExternalCertificateFilePath -Path <Path to Edge PFX Cerfiticate> -Target EdgeServer -Import. Used to both import new certificate to the Edge server and assign it to the external interface. This action will put appliance in maintenance mode.
  • Set-CcExternalCertificateFilePath -Path < Path to PSTN Gateway Certificate > -Target MediationServer -Import. Used to import new certificate chain of the issuing CA for PSTN Gateway certificate to the Mediation Server.
Defined certificate paths saved in "C:\ProgramData\CloudConnector\module.ini":
  • ExternalCertificateFilePath.
  • GatewayCertificateFilePath.

New cmdlets

Certificate Management
  • Backup-CcCertificationAuthority: Backs up the certification authority service to a file and saves it to the CA folder under the site share directory.
  • Export-CcRootCertificate: Exports the root CA certificate to a local file on the Cloud Connector host server.
  • Renew-CcCACertificate: Reinstalls the Certification Authority Service AD Server to create a new root CA certificate..
  • Renew-CcServerCertificate: Renews the certificates for Cloud Connector when they are near expiration or already expired.
  • Remove-CsCertificationAuthorityFile: Removes the certification authority service backup file "<SiteRootDirectory>\CA\SfB CCE Root.p12" in the CA folder under the site share directory for Cloud Connector.
  • Remove-CcLegacyServerCertificate:Removes legacy server certificates on the Central Management Store, Mediation Server, and Edge Server after you execute the Renew-CcCACertificate or Renew CcServerCertificate cmdlets.
  • Reset-CcCACertificate: Resets the certificate authority servers to install a new certificate authority certificate.
CLS Logging
  • Start-CcLogging: The Start-CcLogging cmdlet generates the incoming and outgoing call log for a Skype for Business Cloud Connector Edition instance.
  • Stop-CcLogging: The Stop-CcLogging cmdlet stops generating the incoming and outgoing call log for a Skype for Business Cloud Connector Edition instance.
  • Search-CcLog: The Search-CcLog cmdlet searches the incoming and outgoing call logs in the Skype for Business Cloud Connector Edition appliance log directory.
Cloud Connector cmdlet reference: https://technet.microsoft.com/EN-US/library/mt740652.aspx

1.4.2 Resolves Following Issues

]]> Understanding Cloud Connector Edition Auto Recovery https://docs.microsoft.com/archive/blogs/sfbhybridvoice/understanding-cloud-connector-edition-auto-recovery Fri, 10 Mar 2017 14:53:40 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=75 Auto Recovery Requirements For details on Auto Update requirements, please see Understanding Cloud Connector Edition Auto Update

Auto Recovery Process

Detection: Process for detecting an appliance status runs every 60 seconds and status is updated in the online tenant and cached locally in “CCE Site Directory\Site_EdgeFQDN”. During detection the management switch will connect to the cloud connector virtual machines to run PowerShell session to get the status of the services. Monitoring: The following services are actively monitored:
  • Mediation Server: RTCSRV and MEDSVC
  • Edge Server: RTCSRV
Recovery Process: If any monitored services are detected stopped:
  • The appliance status set to Error, and its network connection disconnected.
  • The AutoMaintenanceStatus value set to 3 indicating Auto Recovery status in the "Appliance Root\CceSevicePersistent" file.
  • Service tries three (3) times to restart the failed service(s). The HAHealthKeepRecover value in the "C:\ProgramData\CloudConnector\ApplianceManager.ini" increments for each attempt.
  • After 3 failed attempts, a process to restart each Cloud Connector Virtual Machine begins in this order: AD, CMS, MS, Edge.
  • The network connection restored when the appliance is running.
  • If recovery cannot complete, the following events logged by the Cloud Connector management service in the Windows application event log:
    • Event ID 304: Completed recovery of the appliance, recover result: CannotRecover
    • Event ID 20006: Appliance cannot be recovered automatically, please recover manually

Manual Recovery

To manually recover the appliance, first review the Cloud Connector management service log for details on what prevented automatic recovery from being successful. Log location: "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService\CceManagementService.log" If connectivity to the Cloud Connector virtual machines fails due to an access denied error message, manually check network connectivity to the machines. If any services cannot be started on either the Edge and/or Mediation server, connect to the virtual machines and review the Lync Server event logs for relevant error messages and try to start the services manually. clip_image001

Reset the Appliance

Once services are running, reset the appliance status to a running state by running the following cmdlets in an Administrative PowerShell on the Host Appliance:
  • Enter-CcUpdate and wait for the process to complete.
  • Exit-CcUpdate and wait for the process to complete.

Confirm the Appliance Status

Confirm the appliance has returned to a running state as follows:
  • Open the "Appliance Root\CceSevicePersistent" file in Notepad and check confirm the value for AutoMaintenanceStatus is 0, and the IsInManualMaintenance value is false
  • Connect to Remote PowerShell and run Get-CcHybridPSTNAppliance and confirm the Status for the appliance is Running.
Note: You can also check the status of appliances from the on premises PSTN tab in the Voice section of Skype for Business Admin Center in your Office 365 tenant portal.

Worst Case Scenario

If the current version of the appliance cannot be recovered, run Switch-CcVersion to switch to the backup version. After the backup version is confirmed running, uninstall the non-working version with: Uninstall-CcAppliance -Version “# of non-working version”. Note that when the backup version is running, there will be no High Availability support due to inconsistent running and Cloud Connector script versions. Update to the current version as soon as possible, either by modifying the auto update schedule, or manually. For manual update instructions see Upgrade a single site to a new version in the Cloud Connector Edition configuration guide. Cmdlets to check versions
  • Installed Cloud Connector script version: Get-CcVersion
  • Appliance running version: Get-CcRunningVersion
]]> Understanding Cloud Connector Edition Auto Update https://docs.microsoft.com/archive/blogs/sfbhybridvoice/understanding-cloud-connector-edition-auto-update Fri, 10 Mar 2017 14:50:01 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=115 Note: Cloud Connector should be viewed as a part of Skype for Business Service. We are constantly improving the service and make changes in the Service and in Cloud Connector. If you do not update the Cloud Connector to the latest release, you might end up in the situation when your Cloud Connector will not work properly. Imagine the situation, Microsoft updated the service, added changes in Cloud Connector code to make it work with the new version of the service, but if you don’t update the Cloud Connector, your telephony will not work. Because of these, Microsoft supports only the latest version of Cloud Connector software. Also, to accommodate Update Window, we support the N-1 version for 60 days after releasing a new version.

Auto Update Requirements

  • Outbound internet access to install, manage, and update Cloud Connector on Host Appliance.
  • Outbound internet access on all Cloud Connector VM’s to download Windows updates, or, access to WSUS server as defined in Cloud Connector configuration file.
  • Skype for Business Online PowerShell Module installed on Host Appliance.
  • CCE Management Service is running on Host Appliance.
  • Group Policy to prevent forcefully unloading user registry at log off (required for 1.4.1).
  • Skype for Business Tenant Admin account.

Initial Setup

When the first Cloud Connector appliance is registered, a PSTN site is configured in the Office 365 Skype for Business Tenant, with the name of the Site as defined in the Cloud Connector configuration file, Auto update is enabled, and default time windows are configured for both the software (bits) and operating system (windows) with corresponding Tenant Update Time Windows that are configured in the tenant as part of this registration process. To view the PSTN site settings, run Get-CsHybridPSTNSite in Skype for Business Online PowerShell. To modify, use Set-CsHybridPSTNSite -Identity <Site Name>. clip_image001[6]

Confirm or Modify the Update Schedule for Hybrid PSTN Site(s)

To confirm the schedule for updating Cloud Connector software, administrators can use Remote PowerShell as noted above, or can view the Hybrid PSTN site update schedule in Skype for Business Admin Center in the on premises PSTN tab of the Voice section. Note that updates will run based on the local host time of the Cloud Connector appliance. Please be sure to confirm that auto update is enabled and that a bits update time window is set to correspond to the maintenance window that you want the updates to run. The time that updates will occur is based on the local time of the Cloud Connector host appliance. For example, if the update time window is set for 11PM and the Cloud Connector appliance is in Amsterdam, the updates will occur at 11PM CET (UTC+1). For details on how to configure update time windows, please refer to Modify the configuration of an existing Cloud Connector deployment.

Cloud Connector Auto Update Process

When auto updates are enabled, the Cloud Connector management service will check for updates during the update time window configured. If updates are found, then the update process will proceed with the update.

Detection

The CCE Management service uses cached update time window information stored in the root of the CCE Site Directory\Tenant_<EdgeFQDN> file when checking if updates run. This file is updated from O365 tenant every 30 minutes. Therefore, if you modify the schedule online, it can take up to 30 minutes for the change to be implemented on the host appliance.

Auto Update Process Overview

  1. The auto update process will run on the schedule set by Tenant Administrator based on local host appliance clock.
  2. Update detection will continue to run for the duration specified in the update time window.
  3. If an update is detected, the update process is invoked.
  4. The appliance will be put in maintenance mode and only one appliance per site can enter maintenance mode. The maintenance mode lock is written to the root of the CCE Appliance Directory\CceSevicePersistent file. Values for the AutoMaintenanceStatus in this file are: 0=None, 1=Bits Update, 2=OS Update, 3=RecoveryMode
  5. Update tasks will run.
  6. Once updates are completed and all services are confirmed running, the Appliance will be taken out of maintenance mode.
  7. Repeat steps for the next appliance in site.

Monitor Update Process

The Cloud Connector management service will log events to the Windows Application log with a source of CCEManagementService and detailed information will be written to "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService\CceManagementService.log". Note: the CCEManagementService.log can grow quite large, so you might want to stop the CCE Management Service, and rename this log periodically. There are plans to modify logging in the future to prevent log growth. If the log file size becomes too large to open in a text editor, you must use a text file splitter to break into smaller segments. You can also see the status of the appliance by running Get-CsHybridPSTNAppliance in Remote PowerShell or by viewing in on premises PSTN tab in the Voice section of the Skype for Business Admin Center.

Bits Update Process

During this process, the running version remains in service, and an interim switch is used to connect to the new VM’s. Once the new version installation is complete and services are confirmed to be running, the old version is drained stopped and the network connections are switched to the new version.
  1. Bits update is detected based on the scheduled time window.
  2. Bits update task is triggered.
  3. The Cloud Connector download site is queried and if a new build is detected, then the update will occur.
  4. The appliance is put in maintenance mode, and the appliance status is updated in the Tenant showing Status of Maintenance, and DeploymentStatus of Upgrading, with the new version and the start time that the update began.
  5. Cloud Connector bits are downloaded.
  6. The CCE management service is stopped.
  7. The Skype for Business Online Cloud Connector edition software is updated which requires uninstalling the old version and installing the new version.
  8. New virtual machines are built from the existing VHDX file. If the VHDX is detected to be older than 90 days, the Install Instance script will log the following warning: SFBServer.vhdx was generated more than 90 days before. Use Convert-CcIsoToVhdx to generate it again and apply windows updates.  Note: It is recommended that a new VHDX be built periodically to reduce the amount of time to perform Windows updates for new and updated Cloud Connector machines. It’s not supported to update the VHDX with Windows update and re-run Sysprep as there are a limited number of times that Sysprep can run on a computer.
  9. Once the deployment of the new Cloud Connector is completed and services confirmed running, the switch to the new version will occur as follows:
    1. Change virtual network connections to new Cloud Connector virtual machines.
    2. Shut down the N+1 version.
    3. Remove N+2 version and delete the virtual disks.
  10. The appliance will be taken out of maintenance mode, and the appliance status is updated in the Tenant to reflect updated Status of running, Version number of new build, and DeploymentStatus of Upgraded.
  11. Detailed logs for the download, upgrade of the Cloud Connector software, new version installation and switch to new build will be written to the Logs folder located in the root of the Appliance directory.

Windows Update Process

Windows update process is performed on the active running version. Therefore, when a windows update is detected, the appliance is drained stopped and put in maintenance mode.
  1. OS update is detected during scheduled time window.
  2. OS update task is triggered.
  3. The appliance is put in maintenance mode, and the appliance status is updated in the Tenant showing Status of Maintenance, and OsUpdateStatus of Upgrading, with start time. clip_image003[6]
  4. The RTCSRV service on Edge and the RTCSRV and RTCMEDSRV services on Mediation server are drained stopped.
  5. OS update PowerShell script is copied to the root of the System drive on all CCE VM’s.
  6. Local windows update service is triggered to check for updates either against Windows Updates Internet service, or the local WSUS server defined in Cloud Connector configuration file.
  7. Updates are installed and a check for virtual machine restart is run.
  8. If a restart is required, all Cloud Connector virtual machines are restarted, then a second check for restart is run.
  9. Once updates are completed on all virtual machines, the updates are run on the host appliance and its restarted.
  10. Once the host has been restarted and no additional restarts are confirmed, the appliance is taken out of maintenance mode, and the appliance status is updated in the Tenant to reflect updated Status of running, and OSUpdateStatus of Updated.

Troubleshooting Auto Update

CCE Management Service Logging Level: If you need more diagnostic logging, you can modify the logging level to verbose for the following two settings in the "C:\Program Files\Skype for Business Cloud Connector Edition\ManagementService\Microsoft.Rtc.CCE.ManagementService.exe.config“ (This will cause rapid log growth): <add name="serviceSwitch" value="Information"/> <add name="powershellSwitch" value="Warning"/> If updates are not running because another maintenance task is detected, check the status of the CCE Appliance Directory\CceSevicePersistent file to determine what task is running. {"AutoMaintenanceStatus":#,"IsInManualMaintenance":false} 0=None, 1=Bits Update, 2=OS Update, 3=RecoveryMode Bits update failed to switch version is logged by the CCE Management Service with following error: CceService Error: 20003: Bits update failed to switch version. Appliance running status: Running, error detail: Failed to drain services with exception: [192.168.213.4] Connecting to remote server 192.168.213.4 failed with the following error message: Access is denied Check the networking status on the virtual machines and be sure there are no duplicate IP’s configured.]]> Prepare for Cloud Connector Edition Release 1.4.2 https://docs.microsoft.com/archive/blogs/sfbhybridvoice/prepare-for-cloud-connector-edition-release-1-4-2 Fri, 10 Mar 2017 08:05:50 GMT https://blogs.technet.microsoft.com/sfbhybridvoice/?p=45 Understanding Cloud Connector Edition Auto Update. If you are running version 1.3.8, refer to the manual update instructions Upgrade a single site to a new version in the Cloud Connector Edition (CCE) configuration guide. Also, make sure to export a new sample Cloud Connector configuration file after 1.4.2 is installed for reference, and update your existing configuration file with two new parameters defined in the Common Section: HardwareType and WSUS Server. If you are having problems installing or upgrading to 1.4.2, see Troubleshooting Cloud Connector Edition 1.4.2 Updates and New Deployments.

Confirm Group Policy is Configured

To ensure Cloud Connector successfully automatically updates, confirm that all Cloud Connector host appliance computers have the local Group Policy (GP) applied as documented in the Prepare your environment for Cloud Connector document. Note: This group policy is not required for appliances deployed with version 1.4.2 or higher.

Remove Forward PAI from Cloud Connector Configuration

Use of Forward PAI has been deprecated in 1.4.2 as it’s no longer required to resolve missing caller ID for outbound calls from Skype for Business Mobile clients which has been resolved with changes in outbound routing logic for Cloud Connector.
  1. Open the “Appliance Root\CoudConnector.ini” file in a text editor and check the value for ForwardPAI.
  2. If the value is False, Remove the entire ForwardPAI section and save the configuration file. clip_image002[8]
  3. If the value is True, indicating you are actively using this for mobile caller ID, wait until 1.4.2 is installed before removing this section in the configuration file. Then, follow the Update Trunk Configuration instructions below.
Note: Run Get-CcApplianceDirectory from the Administrative PowerShell console on the Cloud Connector appliance host to locate the appliance directory if unknown Update Trunk Configuration
  1. Connect to Mediation Server
  2. Open PowerShell as Administrator
  3. Run Set-CsTrunkConfiguration -ForwardPAI $False
  4. Confirm replication to all servers with Get-CsManagementStoreReplicationStatus

Set CCE Account Passwords to Non-Expiring

A bug in prior builds caused Cloud Connector account passwords to expire. If the passwords have expired, then the update will fail. To ensure that the upgrade to 1.4.2 is successful, please do the following:
  1. Log in to the CCE AD machine with the CCE Domain Admin password. Open the AD Users and Computers console from Administrative tools, and set both the Administrator and CCEService accounts to non-expiring passwords.
  2. Log in to each of the other CCE machines with the CCE Domain Admin password. Open Computer Management, and set both the Administrator and CCEService accounts to non-expiring passwords.
  3. On the CCE Appliance Host, open Computer Management, and set both the Administrator and CCEService accounts to non-expiring passwords.

Proxy Configured in DMZ

If you require a Proxy in DMZ for Internet access, confirm the following on the Host appliance.
  1. WinHTTP Proxy settings are configured with your proxy server and a Bypass-list including the "192.168.213.*” network used by your Cloud Connector Managements services to bypass the proxy. Otherwise management connectivity will fail and prevent the deployment of Cloud Connector Edition. Note – match the bypass list to the management subnet that is defined in your CloudConnector.ini file. Sample winhttp configuration command: netsh winhttp set proxy "10.10.10.175:8080" bypass-list="*.local;1.*;172.20.*;192.168.218.*'<local>"
  2. Proxy settings are configured per-machine rather than by user. Otherwise Cloud Connector downloads will fail. This can be done either with registry change or Group Policy setting:
    • Registry: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings] ProxySettingsPerUser dword:00000000
    • Group Policy: Computer>Administrative Templates>Windows Components> Internet Explorer: Make Proxy settings per-machine (rather than per user)
Note: Be sure to configure IE Proxy settings on Cloud Connector virtual machines to allow them to reach Windows Update Servers for operating system updates.]]>