Office Communications Server

Securing OCS with ISA Server

Alan Maddison


At a Glance:

  • OCS 2007 R2 Edge Server roles and topologies
  • Configuring ISA Server in a 3-Leg Perimeter network
  • Understanding OCS certificate requirements
  • Creating a Web listener and a reverse proxy for OCS


Edge Servers
Getting Ready
Configuring ISA Server 2006
A Word about Certificates
Reverse Proxy
Last Words

Office Communications Server (OCS) 2007 R2 provides powerful capabilities that allow you to extend your Unified Communications infrastructure to users outside of your organization. This can have tremendous benefits. Functionality such as Web and audio/visual (A/V) conferencing, for example, can improve an organization's responsiveness and effectiveness in its daily business tasks.

If you choose to deploy OCS functionality to remote users and partners, however, you must complete two important steps. First, you will need to deploy OCS Edge Servers to best practices defined in the Office Communications Server 2007 R2 Security Guide. Second, you will need to provide reverse proxy access to the Edge Servers. In this article I'll take a look at using ISA Server 2006 with SP1 to secure your OCS deployment.

Edge Servers

To communicate with other OCS infrastructures and to allow remote access to your organization's network, you need to deploy one or more Edge Servers in your perimeter network (also known as a DMZ) so that users outside the firewall can gain secure access to your internal OCS deployment. OCS 2007 R2 includes three Edge Server roles, summarized in Figure 1, which also describes reverse proxy functionality.

Figure 1 Edge Server roles and reverse proxy for OCS 2007
Role Purpose
Access Edge Server Authenticated external user access including public IM connectivity, federation, Web conferences, and voice functionality.
Web Conferencing Edge Server External Web conferencing and data collaboration.
A/V Edge Server Any external user access to A/V conferences and point-to-point A/V calls.
Reverse Proxy Group expansion, address book file download, Web conferencing download of meeting content.

Prior to the R2 release of Office Communications Server 2007, Microsoft supported four different Edge Topologies that provided varying levels of sophistication and scalability. In general, the topologies differed in the location of the various Edge Server roles:

  • Consolidated Edge Topology
  • Single-Site Edge Topology
  • Scaled Single-Site Edge Topology
  • Multiple Site with a Remote Site Edge Topology

With the release of R2, Microsoft now require that all roles run on the same server with scalability provided by hardware load balancers (the Network Load Balancer, or NLB, component available in Windows is not supported). A Consolidated Edge Topo­logy is a cost-effective approach that has the added benefit of being the simplest to deploy and administer. This topology works for all organizations, and for the purposes of this article I'll focus on this design.

An important point to remember is that in addition to changes in OCS topology support, there have been some significant changes in the network topologies and technologies supported by R2. However, in R2 the A/V Edge Server role must have a public IP address on the external network interface card (NIC) due to the underlying requirement of the Simple Traversal of UDP through NAT (STUN) protocol.

Though Microsoft has been very adamant about this requirement, many Administrators have ignored the documentation and have attempted to implement the OCS Edge Servers in a Network Address Translation (NAT) environment. If you implement NAT in front of your A/V Edge Servers, your users could experience intermittent connectivity problems and may not even be able to establish a connection.

One important point to remember is that as of R2 Microsoft no longer supports Destination Network Address Translation (DNAT) in an OCS environment. This means that the firewall or Load Balancer must be configured with Source NAT (SNAT) before introducing R2 into your environment. Finally, your Edge Servers should not be members of an Active Directory domain.

If the Edge Servers are not configured correctly, you will only be creating more of a headache for yourself when ISA Server is thrown into the mix. Review, validate and test your Edge configuration before moving on.


Figure 2 3-Leg Perimeter deployment of Consolidated Edge Servers

Getting Ready

To deploy the OCS Edge Topology, I'll focus on a common implementation scenario in which a firewall, ISA Server in this case, is deployed in a 3-Leg Perimeter network; a simple, cost-effective approach that shares core concepts with more complex firewall topologies. In this design, the three legs correspond to the internal network, the perimeter network, and the external (Internet) network. This approach not only allows external user access to the OCS infrastructure, it also supports using ISA Server in a reverse proxy role.

Figure 2 depicts a logical view of an ISA Server 3-Leg implementation. Before beginning to configure ISA Server, it is a good idea to have already laid out the IP addressing and Fully Qualified Domain Name (FQDN) mappings for your Edge Servers and ISA Server. This will make the configuration process a lot simpler and quicker.

Figure 3 shows an example of a suitable check list. Note that I have used a private IP address range for informational purposes only; you will have to use public IP addresses.

Figure 3 Sample configuration
Role Fully Qualified Domain Name Public IP Address NIC External User Port Requirements Internal User Port Requirements
Access Edge Server External 5061,443 5061
A/V Edge Server External 443,3478 50,000-59,999 443, 5062 50,000-59,999
Web Conferencing Edge Server External 443 8057

Figure 4 shows the IP addressing information for the ISA Server, including the IP address used to publish (reverse proxy) the OCS Address Book and Content Web site used by the Web Conferencing Server.

In a Consolidated Edge Deployment, there is typically a single physical server with two NICs, one for internal traffic and one for external traffic. The NIC connected to the corporate network (internal) will have an IP address from your existing internal IP address range. The NIC that connects to remote users (external) will use at least one public (fully routable) IP address for the A/V Edge Server. Although not strictly necessary, it is simpler to also assign public IP addresses to the other Edge roles, as in Figure 4.

Figure 4 ISA Server IP addresses
ISA Interface Fully Qualified Domain Name IP Address
External N/A
Reverse Proxy
Perimeter N/A
Internal N/A

You will also need to create unique subnets for your public IP address space. ISA Server will need two IP addresses on its external interface (one of which will be used for the reverse proxy), and you will need four public IP addresses for your perimeter network (three for the Access Edge Servers and one for the ISA Server perimeter interface). As an example, this is shown in Figure 3 as 29-bit subnet mask (, which would give you six usable IP addresses per subnet.

If you have been allocated only a small number of public IP addresses, you can use network address translation for the Access Edge Server and the Web Conferencing Edge Server roles and connect the A/V Edge Server directly to the Internet. However, this approach is a little less secure in that you are bypassing the packet inspection capabilities of ISA Server and would require a third network interface card for your Edge Server.

Configuring ISA Server 2006

To configure the 3-Leg Perimeter, you need to launch the ISA Server management console and select Networks under the Configuration node in the left-hand pane, as shown in Figure 5. Next, click on the Templates tab in the right-hand pane and select the 3-Leg Perimeter template to start the Network Template Wizard.

This process will erase any existing configuration and rules, so note that if you are not using a new system, you should take care to export the configuration, an option offered by the wizard on the second screen. Clicking Next on the second screen begins the configuration process by asking you to input the IP address ranges that define your internal network.


Figure 5 Selecting 3-Leg Perimeter template

You have a number of options to help define the internal address space. If your organization is small, simply adding the network adapter that is connected to the corporate network is sufficient. Larger organizations will need to use the other options, such as adding IP address ranges to fully define their internal address space.

After entering this information, click Next to move to the next screen to define the perimeter network address space. It is usually sufficient to add the adapter you have dedicated to the perimeter network. Next, you must select a firewall policy. Be sure to choose the "Allow Unrestricted Access" firewall policy, then click Next to go to the Summary screen where you can review your configuration. Select Finish to complete the wizard and then, to accept these changes, click Apply and then OK.

The basic 3-Leg Perimeter is functional at this point but it needs some additional configuration in order to permit network traffic between the OCS Edge Servers and external users. The first thing to note is that the default configuration of a 3-Leg Perimeter in ISA Server includes support for VPN users. If you don't need this type of remote access, remove this item.

To do so, select Firewall Policy in the left-hand pane of the ISA Management console and then right-click the rule called VPN Clients to Internal Network and select Delete. To accept the changes, choose Apply and then OK. Two rules remain: the Unrestricted Internet access rule and the Default rule.

The next step is to add the computer objects that represent your Edge Servers. Select Firewall Policy and then choose the Toolbox tab, as shown in Figure 6.


Figure 6 Firewall Policy rules

Click on the New button and select Computer from the dropdown menu. Add the name of the Edge Server, for example Access Edge, and then enter the IP address for that server. Repeat this step for the A/V Edge Server and the Web Conferencing Edge Server. When you are finished, you should see your three computer objects listed under the Computers container.

The next step is to add the protocols used by OCS to the ISA Server configuration. You will need to add two new protocols, as shown in Figure 7.

Figure 7 Protocols used by OCS
Protocol Name Protocol Type Protocol Direction Port Range
Mutual Transport Layer Security (MTLS)/Session Initiation Protocol (SIP) TCP Outbound 5061-5061
Simple Traversal of UDP through NAT (STUN) TCP Outbound 50,000-59,999
UDP Send *50,000-59,999
UDP Send 3478-3478

Note the asterisk in Figure 7. In OCS 2007 R2, this port requirement is necessary only if you use the A/V capabilities of OCS with a Federated Partner running Office Communications Server 2007. Remote users will not need these ports to be open.

It is worth emphasizing that it is a security best practice to only open ports that you require. For example, consider A/V conferencing with a Federated Partner, which requires UDP ports in the range 50,000–59,999. If you don't have a Federated Partner then you don't need to open these ports.

In the ISA Server management console, within the Firewall Policy and the Toolbox tab selected, click on Protocols, then on New and on Protocol to launch the New Protocol Definition Wizard. When the wizard launches, enter a name for the protocol, then click Next to go to the Primary Connection Information screen. On this screen click New and then add the information for the MTLS/SIP protocol, as shown in Figure 7.

To finish adding the protocol, click OK and then the Next button twice; you do not need to configure anything on the Secondary connections screen. Now click the Finish button on the Summary screen, and repeat these steps for the STUN protocol. Make sure you apply these changes to ISA Server when you have finished adding the protocols.

The Persistent Shared Object Model (PSOM) protocol (a proprietary protocol for transporting Web conferencing content) is not listed in Figure 7. This is because PSOM is used for traffic between the Web Conferencing Server and the Web Conferencing Edge Server. This traffic is sent out on the internal network card of the Edge Server.

After adding the above two protocols from Figure 7, the next step is to create the three Access rules that will permit users to connect to the Edge Server from the Internet. Once again, having this information at hand will make these configuration steps a lot simpler and less prone to error. Figure 8 shows the information you will need to create the three external access rules.

Figure 8 Access rule configuration
Access Rule Name Rule Action Protocols Access Rule Source Access Rule Destination User Sets
Access Edge Allow HTTPS MTLS/SIP External Access Edge All Users
A/V Edge Allow HTTPS STUN External A/V Edge All Users
Web Conferencing Edge Allow HTTPS External Web Conferencing Edge All Users

Start by selecting Firewall Policy and the Tasks tab in the ISA Server Management console and then click on Create Access Rule to launch the New Access Rule wizard. You can create the rules in any order, so let's start with the Access Edge rule. Once the wizard launches, enter a name for the rule and click Next. On the Rule Action screen, make sure that the Allow radio button is selected and click Next.

The following screen requires you to add the Protocols to which the rule applies. Click the Add button on the right-hand side of the screen to launch the Add Protocols window. Under the Common Protocols folder, select HTTPS and press Add, then select the STUN protocol—which should be listed under the User-Defined folder—and click Add. Click Close and then Next to move onto the Access Rule Sources screen.

This step requires you to select the Access Rule source, and in the case of the Access Edge rule, you will need to select the External network object by clicking Add and selecting it from the Networks folder. Then click Close and then Next to move to the Access Rule Destination screen. On this screen click Add and select the Access Edge computer object you created earlier from the Computers folder. Click the Close button and then Next to move to the User Sets screen.

Leave the All Users set default selection and click Next. Review the Summary, then click Finish to complete the process. You can now create the two remaining access rules using the information you compiled based on Figure 8. Make sure you apply these changes to ISA Server once you have finished.

A Word about Certificates

The last steps in this configuration process are focused on creating the SSL listener and creating the reverse proxy (publishing a Web application) for Office Communications Server. It's important to understand the certificate requirements of Office Communications Server and how these requirements impact ISA Server configuration.

The most important point to note here is that Office Communications Server requires the use of x.509 certificates and does not support wildcard certificates in the Common Name (Subject Name). You will need to specify a Subject Alternate Name (SAN) when requesting the certificate for the Edge Servers.

Since ISA Server 2006 SP1 introduced full support for SAN certificates, this will not present any problems. With this in mind, you need to address the certificate requirements for both the internal and external interfaces of the Edge Server. If your Active Directory environment does not use a top-level domain namespace, you will need to use a certificate from a trusted third-party Certificate Authority (CA) for the external interface and a certificate from an internal enterprise CA for the internal interfaces. In this scenario, the Root Certificate for your enterprise CA must be installed on clients that will access OCS internally as well as on the ISA Server; this ensures that the necessary trust relationships are in place for the OCS configuration. The external third-party certificate should already be trusted, and therefore the Root Certificate of the third-party CA doesn't need to be installed on all clients.

The certificate required for the reverse proxy configuration corresponds to the FQDN of the OCS address book and conference content download. In Figure 4, this was but can obviously be anything you choose as long as the certificate comes from a trusted third party. Of course, your SAN certificate from this third party would also include the FQDNs of your Edge Servers, samples of which you can also see in Figure 3.

Reverse Proxy

The first step in creating a reverse proxy for OCS is to create an SSL Web Listener. To do this, click on the Firewall container in the left-hand pane and select the Toolbox tab in the right–hand pane. Select Network Object from the list, click the New button and select Web Listener from the dropdown menu. This will launch the New Web Listener wizard, which prompts you to enter a name for the Web Listener.

The name should be meaningful to you, but it doesn't necessarily have to be associated with OCS as the Web Listener could potentially have multiple uses. When you click Next, the Client Connection Security screen appears with the default option selected—Require SSL secured connections with clients. Leave the selection as is and click Next to move to the Web Listener IP Address screen, where you should check the External networks option. Next, click the Select IP Addresses button and choose the IP address that you have assigned to the reverse proxy (see Figure 4).

After clicking OK and then Next, you will see the Listener SSL Certificates screen. Select the option to Assign a certificate for each IP address and then click the Select Certificate button. Now select the certificate associated with the OCS address book and Web conference content that was discussed earlier. After selecting the certificate, click the Next button to configure Authentication Settings. Make sure that No Authentication is selected in the dropdown box and then click the Next button twice. Finally review the Summary screen and click the Finish button to complete the creation of the Web Listener.

Once the Web Listener is created, you can create the Web Site Publishing rule. To do this, click on the Firewall container and select the Tasks tab, then click on Publish Web Sites to launch the New Web Publishing Rule Wizard. The first piece of information to enter is the name of the rule, so enter a meaningful name such as OCS Content and click Next. On the Select Rule Action screen, select Allow and click Next to move to the Publishing Type screen where you will select Publish a single Web site or load balancer and then click Next.

To configure Server Connection Security, select the option to use SSL, then press Next to proceed to the Internal Publishing Details screen. Here you need to enter the internal name of the Web components server. Click Next and enter the path information, which should be /* to allow all paths. Click Next and enter the public name you created for the Address Book and Web Conference content downloads (in the example here it is Leave the other selections at the defaults and click Next.

Now select the Web Listener you created and click Next. On the Authentication Delegation screen, the dropdown box should read No delegation, but client may authenticate directly. Click Next and confirm that All Users is selected on the User Sets screen, then click Next again and then Finish to complete the process. The configuration of ISA Server is now complete.

Last Words

Extending Office Communications Server to support remote users and partners can yield tremendous benefits but requires a great deal of planning, particularly when it comes to securing your Edge Servers. ISA Server 2006 provides a robust, flexible, and scalable solution that is the ideal platform for securing Office Communications Server 2007.

Alan Maddison is a Senior Consultant specializing in Microsoft technologies with Strategic Business Systems, a division of Brocade.