The final principle is to get your core Office 365 traffic to Microsoft as efficiently as possible; this means minimising the work done on the traffic or ensuring any work done doesn't cause an impact to that traffic. As already noted, when moving to the cloud it's necessary to rethink about how you handle traffic leaving your managed network for trusted endpoints such as Office 365.

The bulk of an enterprise's cloud traffic will be user-initiated connectivity to the cloud and that is what this page is referring to, where the traffic is from the enterprise to Microsoft endpoints which will account for the majority of the traffic flow. For inbound flows, (Microsoft initiated inbound to the enterprise) it is understood that a higher degree of inspection/control may be desired, the volume of traffic in this direction is also comparatively low.

When there is a move to SaaS services like Office 365, numerous connections which were once kept within the corporate network in their entirety, now traverse the egress equipment to reach the endpoints in the cloud. As noted, this is often a large volume of traffic which puts terrific load on the standard egress equipment/method to egress to the internet, and thus is performing break and inspect on all traffic.

Understandably the default security model will often be to break and inspect all outbound traffic to ensure that the endpoint is something we want the users to be able to access, and also that there is nothing malicious being bought into the corporate network, or anything exfiltrated.

Whilst necessary, this inspection is computationally expensive and can impact performance, however, when visiting unknown websites this impact is entirely acceptable to the business. With SaaS services like Office 365 this is certainly not the case and can severely impact user experience whilst requiring massive spending to scale up and still not solve all of the issues which arise.

Here is the question we need to ask ourselves though. Does the same policy which applies to unmanaged and unknown endpoints on the internet, need to apply to known, managed and trusted endpoints used for business-critical services? The answer is hopefully no. Services like Office 365 should be treated somewhere between how you treat internet traffic, and how you treat traffic to your on premises datacentres (which usually has no security applied).

Where exactly in-between depends on how much you trust the endpoints/service in question but it's certainly worth assessing whether what you're doing at the edge is already done, or can be done in the backend and thus is unnecessary inline where it can cause performance issues.

Data Loss Prevention can be done in Office 365 itself and AV/anti malware scanning is also done by Office 365 against certain elements so should not be required inline for them. We're not always going to unknown and untrusted/unmanaged endpoints with Office 365 like we are with the internet traffic, we're going to known, business critical, managed and security controlled endpoints. Microsoft Trust Center holds all the relevant information on how we secure your data in the cloud.

Therefore, Microsoft strongly recommend that SSL interception/DLP etc is not performed on the core Office 365 traffic. Previously it was very difficult to identify which traffic to apply this advice to as Office 365 consists of various elements, some of which are Microsoft owned and IP provisioned, others (CDN, CRL endpoints etc) are not. As described in part one of this series, Microsoft now mark endpoints in one of three categories "Optimize", "Allow" & "Default" which allows us to selectively treat certain traffic elements in different ways.

The advice is that at the very minimum, the 'Optimize' marked endpoints should be sent directly to their destination with no proxy or SSL break and inspect, egressing as close as possible to the user's location, essentially following the principles outlined in this set of posts to the letter.

Any other endpoints (eg the Allow group) if proxying/interception is done, it should be carefully managed so as not to cause a bottleneck. "Default" marked endpoints can be treated like normal web traffic without issue.

This brings us to the next point, what is Microsoft's recommended egress model? How do we recommend you get Office 365 traffic out of the corporate network and to Microsoft? There are essentially three main methods:

• Direct Routing
• Proxied Access
• ExpressRoute (which is essentially direct routing via another path)

There are distinct pros and cons to each method and they may or may not be applicable depending on the type of traffic.

Direct Routing

Direct routing would be similar to that which you have at home, a single TCP session is used to connect to the endpoint with (in most cases) the source IP simply being translated from an internal (e.g. 10.x.x.x) to a publicly routable one on the way out of the managed network. The egress device may also ensure that the destination IP and or port is also allowed. This means the endpoint connected to receives the request from the translated (public) source IP, but the client connects to the public IP address of that endpoint.

This method is generally the recommended way to connect Office 365 services if possible.

Pros:

•    Allows direct UDP traffic meaning Skype can work at its best.

•    Generally, no interference with payload at egress meaning optimal connectivity for all services

•    Allows for local egress use in most cases meaning minimal latency to the service front door and Microsoft's global network

•    Minimal work done on traffic means scaling (whilst still necessary for the volume of connections and network address translation) is less demanding

•    Best connection method to Office 365 core endpoints for most customers when ISP routing is optimal

Cons:

•     It is required to authorize Office 365 URLs/IPs and open required ports on all firewalls used (if controlled egress is desired). These need to be monitored and firewalls updated with changes as missing updates to IP

ranges can cause connectivity issues. However, the Office 365 web service should allow for easy automation of these updates on supporting network equipment.

• Having controlled egress by using the IPs for Office 365 also means endpoints where we cannot provide the IPs (such as CDNs, DNS, CRL lookups) have to be routed via another path with URL based or unrestricted access.

•     Routing to the appropriate egress needs to be managed internally and needs to include external IP address routing (which can be an issue for some customers)

•     External DNS resolution is required (which can be an issue for some customers)

•     Devices still need to scale to the increased connection count needed for Office 365 services

Due to the efficient, low impact manner of egress, allowing connections to flow direct, using the protocol of choice, this method is the recommended method to connect your Office 365 services for the Optimize marked endpoints and ideally the Allow marked endpoints too.

Proxied Access

This is a common method used to connect to the internet as it simplifies the connectivity process and allows a centralized device to control access and intercept traffic.

Pros:

• Easy to Configure to get Office 365 connected
• Often the existing internet access method
• Small Number of IP addresses for clients to direct traffic to
• Uses known ports for easy firewall traversal
• No need to route external IP address on internal network
• Easy monitoring/auditing
• Provides a Security Barrier between clients and the internet

Some of these pros mean a proxy is the only way a customer can access the internet without major network redesign work. All Office 365 services will work though a proxy, even Skype for Business, however there are a large number of drawbacks to this method.

Cons:

• Proxies generally do not handle UDP traffic & Skype traffic is therefore forced over TCP to traverse the proxy.
• Skype's coping mechanisms for poor networks are drastically reduced when TCP is used
• The proxy functions can delay frames on their way through adding jitter and latency
• Older proxies often struggle to deal with the long lived, high throughput connections SAAS services entail.
• Proxies commonly alter TCP level settings which can cause performance issues
• SSL issues can also occur as the proxy is a 'man in the middle'
• Often don't scale and were not installed/designed with SaaS services in mind.
• Capacity upgrades to cope with the additional workload SaaS services will add in terms of Ports/Memory/Processing are likely to be very expensive.
• End result is often poor-quality calls and performance

As you can see, there are some considerable downsides to standard proxies when it comes to SaaS services. Skype traffic is highly likely to run into issues via standard on premises proxies due to the use of the non-optimal (for real-time voice/video traffic) TCP protocol (as opposed to UDP) and the processing at the proxy layer is likely to introduce issues such as jitter. Imagine the load the proxy would be under handling thousands of real-time TCP media sessions during an all hands call for example, scaling up to handle this is likely to be very expensive and still run the risk of causing performance issues due to the protocol used. The other thing to bear in mind is that proxies were likely designed for access to transient endpoints, in that a TCP connection will be made to a website, the data will be obtained and then the session closed and the resources (memory, processing, ports) will be returned to the pool. SaaS services tend to work very differently however. Outlook as an example will open multiple TCP connections per users, and sit there all day with them in use, as such the resources aren't returned to the pool as they would be with transient access and again, the devices need upgrading to deal with this extra load. We recommend around 2000-4000 clients per public IP address for network address translation.

Due to the high risk of these devices causing performance issues due to their design and role being for a different purpose, Microsoft recommends you don't use these types of proxy solutions for the Optimize marked endpoints and ideally the 'Allow' marked ones unless absolutely necessary. If there is no other option, or it is a very strong requirement to use proxies, the following advice should be followed.

•    Ensure the devices are scaled up to cope with SaaS services, in terms of memory, processing and NAT capability

•    Avoid overly centralized proxies which can increase latency

•    Ensure they are in the local region of the client

•    Evaluate Cloud Proxy nodes if the above isn't possible as these often can allow for localized, scalable proxy use

•    Avoid packet inspection (i.e. SSL break & inspect)

•    Ensure all settings are checked and optimized

•    Avoid using Skype for Business through these devices unless they can bypass for UDP

Whilst not a recommendation for any vendor over another, Microsoft are working with various vendors such as Z-scaler and Bluecoat to help better align cloud proxy products to best practices for Office 365. Zscaler for example have a button which automatically optimizes Office 365 traffic (e.g. disables SSL offload) for customers who use the service.

If a proxy is a requirement for your business, it's worth checking that your current implementation is going to work well with SaaS services like Office 365. If not, then it's worth talking to the vendors you choose to remediate this, around their alignment to the Office 365 (and cloud in general) connectivity principals discussed in this post, and ensure they are followed upon implementation.

A final point on proxies, they are absolutely a supported method for our customers to reach Office 365, however they are very likely to provide performance issues if not redesigned & uplifted from their old internet access design, to their new usage with cloud services.

ExpressRoute

ExpressRoute is private peering with the Microsoft global network described above. Essentially, it's simply a private network connection from the edge of the customer network to the edge of Microsoft's network (the same network you'd reach over the internet) avoiding the leg which the internet takes in connecting to Microsoft. This private network can carry some elements of Microsoft bound traffic via two types of peering:

1.    Azure Private – Connecting to virtual networks in Azure (e.g. to private IP addresses on virtual machines)

2.    Microsoft Peering – Connecting to Azure public IP ranges and a subset of Office 365 endpoints (the same public endpoints reachable over the internet path)

These two types of peering are the recommended connection methods to Azure endpoints, they are relatively easy to configure requiring little change in the corporate network infrastructure. The Office 365 elements of Microsoft peering is the area we're interested in with regards Office 365 connectivity, this type of peering requires authorization from Microsoft to enable, and this is for good reasons, however let's look at the pros of this type of peering first.

Pros:

•    We can provide a 99.95% SLA for availability

•    Because it's a dedicated circuit for Microsoft traffic and managed end-to-end, it can provide predictable performance and bandwidth

•    It can provide better QoS capability than the internet path for Skype (All QoS markings are stripped at the edge over the internet path).

•    It avoids the internet path for the bulk of Office 365 traffic, some organizations have a regulatory requirement for this.

•    If configured such, it allows a customer to bypass network egress equipment doing SSL interception or other behaviour which may cause a bottleneck

•    It allows Skype for Business to use UDP which is the preferred protocol for performance on real-time traffic

Cons:

•    Performant and working internet connectivity is still required for endpoints which cannot use ExpressRoute (e.g. DNS, CDN, CRL checks) As such if this internet pipe is not available, Office 365 is largely inoperable.

•    A well configured internet connection can, in many cases, give a similar, or in some cases, better performance levels (for example if the internet peer point was closer than the nearest ExpressRoute peer point).

•    Often encourages the hub and spoke model for connectivity which runs contrary to the high-level guidance of local connectivity.

•    Often a higher cost of implementation and usage than a standard internet connection. (This isn't always the case depending on the locality and equipment upgrades required)

•    Enabling this type of peering (Microsoft as opposed to Azure) is very complex and without what we see as typically 2-6 months of planning and work from a large cross skilled team, will very likely result in an outage of your Office 365 implementation

•    If the direct method is used (i.e. non-proxied to the edge) then external DNS and IP routing needs to be available as with the direct internet path above.

•    It is often possible to resolve performance issues quicker, easier and at a lower cost by isolating the issue and resolving it via using an unhindered, direct internet peered connectivity model or other optimized method.

•    Investment in your internet egress is likely to be beneficial to other cloud elements outside of Office 365

Because of this list of cons, especially the complexity and high risk of outages if not correctly implemented, Microsoft have a review policy of requests to use ExpressRoute for Office 365, this is so we can discuss these pros/cons with the customer and ensure that all parties are aware of the 2-6 months of planning, extra complexity and what ExpressRoute can/cannot deliver. The end goal is that if the customer chooses ExpressRoute they have done so fully armed to make an informed decision that it's the right thing for their business. Also, that they are aware of the guidance to ensure the implementation is a success and is going to deliver the desired benefits before spending the time, money and effort implementing.

We have a wealth of technical guidance which covers the implementation, routing, some training videos and some Ignite 2016 content which should give you a good overview of what is required to implement this type of peering. As you can see, this option is not for everyone, in fact a direct internet path connection is the best for the majority of use cases. However, if you think ExpressRoute might be the right thing for your organization's Office 365 traffic, after reviewing the links here, contact your Microsoft account team for assistance in requesting a review for approval and we can then work to help you make the right decision for your business.

So, in summary, there are an array of ways to get your traffic to Microsoft's network, direct networking generally works best for the core endpoints to ensure unhindered and quick access, but where you need to use a proxy, ensure it isn't causing a bottleneck to your traffic and follows the principles outlined in this post, and finally ExpressRoute for Office 365 is not simple to implement and isn't for everyone.

The third principle is to Optimize Route length and avoid network hairpins.

In the majority of cases, the shortest path the Microsoft global network will offer the highest levels of performance to your users for the service. As we discovered in part two, this is due to the global availability of both Microsoft's global network to transport your data in addition to the availability of local service front door elements for Office 365.

A network hairpin would be we use the WAN for a long distance to route traffic back to a centralized location to egress to the internet there to then route out to an Office 365 endpoint which is likely available near the user. Or similarly, using a VPN for remote users to route traffic back into the corporate network to egress to the internet via the security stack there.

Take the diagram below as an example. We have hairpins occurring both for the branch office location which hairpins traffic through the Head Office egress, and also the remote users are forced to tunnel in through a VPN concentrator into the corporate network, then out through the standard egress to reach Office 365 services. In both cases, the likelihood is that users in these locations will experience suboptimal performance with the service in addition to considerable costs for Contoso to deliver this model.

Even routing Office 365 traffic into Azure (via a proxy hosted there for example) would be considered a hairpin as there is a good chance the endpoint we're talking to may not live within the datacentre but out on the network edge, traffic will then hairpin into Azure and back out to the network edge before having to be routed back into the datacentre again. There are also numerous Office 365 endpoints which do not live in Microsoft's infrastructure such as CDN, CRL, DNS endpoints which will also end up hairpinning.

Incorrect DNS configuration could also cause a hairpin. As we learnt in the previous page, DNS needs to be resolved locally so as to find a local service front door. Performing DNS lookups at a different location to the egress risks traffic hairpinning to where the DNS call was completed to hit the service front door. As an example, if I egressed in Sydney Australia but DNS was resolved in NYC, my Outlook client would be directed to a CAFE server in the North America region rather than one in Australia.

Whilst Microsoft's strongly recommended best practice is to send at least the Optimize marked endpoints direct to the service, care should also be taken if choosing a vendor in the cloud based security/network access field if applicable. If this model is in use take great care to ensure your chosen vendor is able to provide infrastructure close to your users. For example, performance is likely to be suboptimal if my cloud proxy vendor's nearest nodes are in Singapore but my users are in Sydney. Also check how well they peer onto Microsoft's infrastructure i.e. that this is done locally to their resources. In essence, it's essential to check how well the provider conforms to the connectivity principles outlined in this article series, for example, how well they allow you to differentiate the Office 365 traffic from normal web traffic? (see part 1).

Moving on to optimized route length. If we've delivered direct, local egress for our users for their Office 365 traffic, how do we test the route length is optimal and my ISP is peering well with Microsoft's network?

Using a simple tracert to known endpoints we can look at both the latency and routing path to Microsoft's network. Tracert will only work when there is a direct path to the endpoint, so if on a corporate network where a proxy is used to egress the traffic (or ICMP is blocked) then we'd have to run this action at the edge of the corporate network, in front of, or on the proxy.

Test endpoints can be as per the following:

Service: Outlook/Exchange Online

• Endpoint to test: outlook.office365.com
• tracert outlook.office365.com

1 1 ms 1 ms 1 internal router 1 [10.16.9.2]

2 1 ms 1 ms 1 internal router 2 [10.3.7.17]

3 3 ms 2 ms 2 internal router 3 [10.3.7.7]

4 2 ms 2 ms 2 Internal router 4 [10.3.7.12]

7 3 ms 3 ms 3 ISP router 1 [194.x.x.x]

8 4 ms 3 ms 5 ISP Router 2 [194.x.x.x]

10 4 ms 6 ms 4 ms ae26-0.icr01.lon24.ntwk.msn.net [104.44.239.105]

11 13 ms 13 ms 14 ms be-120-0.ibr02.lon24.ntwk.msn.net [104.44.21.115]

12 13 ms 13 ms 13 ms be-8-0.ibr02.dub08.ntwk.msn.net [104.44.17.90]

13 13 ms 13 ms 13 ms ae120-0.icr01.dub08.ntwk.msn.net [104.44.11.86]

14 12 ms 12 ms 14 ms ae22-0.db3-96c-3a.ntwk.msn.net [104.44.238.38]

16 17 ms 13 ms 14 ms 104.146.132.25

In the above tracert taken in the UK on a corporate network you see the connectivity hit Microsoft's backbone (msn.net) in LON24 (London) in 4ms, seen in hop 10. So peering here is excellent, and in an expected place. Further we can see the CAFE server we are using at this point in time is in Dublin (db3), seen in hop 14.

Service: SharePoint/OneDrive for Business

• Endpoint to test: tenantname.sharepoint.com (eg contoso.sharepoint.com)
• Tracert contoso.sharepoint.com (changing contoso to your tenant name)

1 <1 ms <1 ms <1 ms – Local gateway 192.168.1.1

4 6 ms 6 ms 6 ms ISP Router 1

5 8 ms 6 ms 6 ms ISP Router 2

7 12 ms 9 ms 10 ms ISP Router 3

8 16 ms 15 ms 9 ms msft-decix-02-dxb30.ntwk.msn.net [185.1.8.32]

9 12 ms 11 ms 11 ms ae26-0.icr01.dxb20.ntwk.msn.net [104.44.233.49]

10 40 ms 40 ms 40 ms be-120-0.ibr02.dxb20.ntwk.msn.net [104.44.20.235]

11 40 ms 39 ms 39 ms be-7-0.ibr02.bom30.ntwk.msn.net [104.44.18.205]

12 39 ms 44 ms 39 ms ae21-0.bom01-96cbe-1b.ntwk.msn.net [104.44.23.246]

13 * * * Request timed out.

14 * * * Request timed out.

15 39 ms 40 ms 39 ms 13.107.136.9

In this example taken in Dubai UAE we see the peering occur on Microsoft's new peering location in Dubai (DXB30) in hop 8. As we don't have a full set of Office 365 services live in the Dubai region at the time of writing, the AFD endpoint being used is bom01 in hop 12 (Mumbai). Again, this peering is in an expected location (see the table in the previous part to see where peering is available).

If we trace to the same endpoint from a machine the UK we see the following:

Tracing route to spo-0004.spo-msedge.net [13.107.136.9]

1 1 ms 1 ms 1 ms network gateway - [192.168.1.1]

3 16 ms 23 ms 13 ms winn-core-2b-xe-030-0.network.virginmedia.net [62.253.120.137]

6 25 ms 20 ms 20 ms tcl5-ic-5-ae0-0.network.virginmedia.net [62.254.84.66]

7 21 ms 19 ms 17 ms ae23-0.lts-96cbe-1b.ntwk.msn.net [207.46.219.212]

8 22 ms 21 ms 25 ms ae25-0.icr02.lon22.ntwk.msn.net [104.44.239.103]

9 22 ms 18 ms 20 ms ae28-0.lon04-96cbe-1b.ntwk.msn.net [104.44.239.120]

10 21 ms 21 ms 21 ms ae20-0.lon21-96cbe-1b.ntwk.msn.net [104.44.238.202]

13 21 ms 26 ms 23 ms 13.107.136.9

Above you're seeing the ISP peer with Microsoft in London (LTS/LON22) which is also where the AFD endpoint resides. If we trace to the same endpoint from the same location via a different ISP then we also see this ISP peering in London.

Tracing route to spo-0004.spo-msedge.net [13.107.136.9]

1 1 ms 4 ms 3 ms Network Gateway [192.168.1.1]

4 7 ms 7 ms 7 ms 31.55.186.180

5 12 ms 8 ms 8 ms 195.99.127.88

6 11 ms 9 ms 8 ms peer2-et0-0-0.slough.ukcore.bt.net [62.172.103.200]

7 11 ms 9 ms 9 ms 109.159.253.203

8 11 ms 10 ms 15 ms ae3-0.lon21-96cbe-1a.ntwk.msn.net [198.206.164.68]

13 9 ms 9 ms 10 ms 13.107.136.9

An important point of note. It shouldn't be expected that the perfect peering location for you is used in every instance, the important thing is that it's in a sensible location. For example, a connection in Berlin in Germany may peer with Microsoft in Frankfurt (despite peering being available in Berlin) as the exact location is down to the ISP's preference and capability. However, an ISP in Berlin should not be peering with Microsoft in NYC for example as this will undoubtedly cause performance issues.

You'll notice most router names have a local code in them to help us identify them, invariably they are the IATA location or airport codes for the area. BOM being that for Mumbai airport (Bombay being the old official name for Mumbai)

Service: Teams

• Endpoint to test: world.tr.teams.microsoft.com
• Tracert world.tr.teams.microsoft.com

Again in this trace, as it's taken in the UK, we can see the peering occuring in London (as you'd expect) and the traffic hitting a Teams relay server in London (lon12)

Tracing route to world.tr.teams.microsoft.com [13.107.64.2]

1 12 ms 18 ms 5 ms Local Network Gateway [192.168.1.1]

3 9 ms 16 ms 16 ms winn-core-2a-xe-133-0.network.virginmedia.net [62.253.121.237]

7 33 ms 23 ms 22 ms tcl5-ic-5-ae0-0.network.virginmedia.net [62.254.84.66]

8 24 ms 18 ms 22 ms ae23-0.lts-96cbe-1b.ntwk.msn.net [207.46.219.212]

9 19 ms 21 ms 19 ms ae25-0.icr02.lon22.ntwk.msn.net [104.44.239.103]

10 21 ms 20 ms 21 ms ae28-0.lon04-96cbe-1b.ntwk.msn.net [104.44.239.120]

11 22 ms 23 ms 19 ms ae3-0.lon21-96cbe-1a.ntwk.msn.net [198.206.164.68]

14 27 ms 32 ms 25 ms world.tr.teams.microsoft.com [13.107.64.2]

As part of the improvements Microsoft announced in 2018 to the way the required URLs and IPs for Office 365 are published, we have also changed how the URLs are categorized to help customers deliver best practice in terms of connecting to the required endpoints for the service.

In the old system, URLs are marked "Required" and "Optional" with a very large list of URLs in each category, published via an XML file.

Traditionally Microsoft advised that customers optimize connectivity to Office 365 for the "required" endpoints. That often means, not using traditional proxy methods at the network edge due to the bottlenecks which commonly occur at this point. This is often due to the load Office 365 puts on these devices. In addition, the work often done at this layer such as SSL inspection for AV scanning or DLP which is often replicated in the application/service itself when it comes to Office 365 endpoints. Proxying traffic also often means that in most cases, Skype for Business or Teams uses TCP for voice/media calls as opposed to its preferred method of using UDP directly, with a very likely drop in performance and quality. More on the challenges with using traditional proxies for SaaS services like Office 365 can be found in another of my blog posts.

The difficulty with this categorization of the URLs (Required & Optional) is that applying this advice has become increasingly difficult for customers as the Office 365 service has grown. The number of endpoints in the "Required" list is considerable and in addition, some do not reside on Microsoft infrastructure (eg CDNs) which we therefore do not provide IPs for. A customer may therefore understandably want to apply SSL inspection on some of these endpoints due to the number and differing nature of them. It's also hard to correlate which URLs have which security features (such as AV scanning/DLP) applicable to it within the service, and thus don't require it at the network edge. The net result is that often key Office 365 traffic ends up being sent via an unoptimized path and encounters performance issues as a result, or it is a large challenge to optimize the full list of endpoints.

We therefore set about a piece of work to analyze all the URLs for the service and identify those which really require optimization to enable customers to focus on the traffic which really requires special consideration, i.e. those which are either very high volume in terms of bandwidth/transactions/connection count and thus put heavy load on proxies, or those which are particularly latency sensitive and require the direct path with minimal interference, for example Skype voice traffic.

This piece of work showed that a very small number actually require the highest levels of optimization, in the order of around 10 endpoints. This small list however accounts for 75-90% of Office 365 Bandwidth, connection and transaction count. These core endpoints are those which will put massive load on traditional proxy infrastructure and/or require that optimized path a direct egress best provides.

With this small list in mind, we can now work around the problem with traditional egress models in a much more precise manner, by concentrating on optimizing the URLs which both cause the problem (in terms of load) and also are the biggest victims of it in terms of performance.

We have therefore categorized the URLs into the following three categories to help customers deal with the endpoints in specific ways according to the needs of the endpoint and the business.

• Optimize for a small number of endpoints that require low latency unimpeded connectivity which should bypass proxy servers, network SSL break and inspect devices, and network hairpins.
• Allow for a larger number of endpoints that benefit from low latency unimpeded connectivity. Although not expected to cause failures, we also recommend bypassing proxy servers, network SSL break and inspect devices, and network hairpins. Good connectivity to these endpoints is required for Office 365 to operate normally.
• Default for other Office 365 endpoints which can be directed to the default internet egress location for the company WAN.

Use of these categories, how they simplify connectivity to Office 365, and what actions you can take to make use of them is detailed in Office 365 Network Connectivity Principles.

Optimize Category

URLs in the Optimize category all have the following characteristics:

• Are Microsoft owned and managed endpoints hosted on Microsoft infrastructure.
• Have IPs provided
• Are routable via ExpressRoute (If the circuit is authorized for this type of traffic).
• Expected to remain low in number of URLs
• Are High volume and/or latency sensitive
• Low rate of change to URLs/IPs compare to other two categories

The Optimize list at the time of publishing is outlined below.

The above (optimize) endpoints will require access to the following IP address ranges:

• Exchange Online
• SharePoint Online
• Skype for business

For all customers, the advice is that, for the endpoints noted above, where possible:

• If proxies are in use, send the above URLs direct in the PAC file and allow a direct path through a firewall/NAT which has an allow list for the IPs and Ports listed.
• Ensure devices and paths handling this traffic can scale to handle the demand in terms of port requirements and bandwidth.
• Bypass or whitelist these URLs on network devices which do SSL decryption, DPI or content filtering, or any other interference of the traffic which could delay it.
• Look at applying the security elements your require for these endpoints, within the service and thus allow the traffic out of the network without hindrance.
• Egress these endpoints as close as possible to the users so the traffic can hit Microsoft's infrastructure locally.
• For remote users ensure these endpoints are sent direct via split tunnelling if VPN solutions are used
• Ensure that DNS resolution for these endpoints is done at the same location as the egress path

The advantage of the above list is we can send a customer's specific, and high-volume OneDrive/SharePoint traffic (tenantname.sharepoint.com & tenantname-my.sharepoint.com) direct, and apply DLP in the service if required, whilst sending any other tenant traffic (*.sharepoint.com) which may be required for sharing/collaboration, via an inspecting proxy.

Whilst most customers do not require it, if strict control of which tenants are accessible then tenant restrictions can be implemented to control which tenants can be accessed from within the corporate network.

By optimizing this list of endpoints, you can ensure that the traffic which is likely to put the most load on your network infrastructure, and/or is latency sensitive is treated in such a way that the service performs at its best, whilst removing the risk of overloading your standard network egress.

Allow Category

Access to the URLs marked as 'Allow' are required for Office 365 to function, however they are not as sensitive to network performance and latency as those marked as 'Optimize'. The amount of bandwidth these endpoints will consume is a small proportion of the whole figure and also connection counts will be lower than the optimize category.

Allow endpoints have the following characteristics:

• Are required for the service to function
• Are dedicated to Office 365 and hosted in Microsoft datacenters.
• Are not as highly sensitive to network performance as the optimize category
• Are not comparatively high bandwidth or connection count endpoints
• Higher rate of change than the optimize category
• Not all endpoints in this category will have IPs provided

Whilst these endpoints are required for the services to work and aren't as sensitive as the Optimize URLs, they still are critical to the running of the service. A heavy delay to connectivity to one of these endpoints could cause performance issues for users. As such it is recommended that these URLS are optimized as far as possible.

Recommended advice in an ideal situation is as follows:

• If possible, bypass or whitelist Allow endpoints on network devices and services that perform traffic interception, SSL decryption, deep packet inspection and content filtering. If it is not possible to bypass these URLs, ensure the work is as light touch as possible, and on optimized, well scaled devices which minimize any delay.
• If proxying this traffic, ensure proxy authentication for these URLs is disabled to remove any delay that may cause.
• If possible, prioritize the evaluation of these endpoints as fully trusted by your network infrastructure and perimeter systems.
• Egress these endpoints as close as possible to the users so the traffic can hit Microsoft's infrastructure locally.
• Ensure that DNS resolution for these endpoints is done at the same location as the egress path
• Prioritize these endpoints for SD-WAN integration for direct, minimal latency routing into the nearest Internet peering point of the Microsoft global network.

   

  Default Category

   

  This final category represents endpoints for Office 365 services and dependencies that do not require any optimization and can be treated by customer networks as normal Internet bound traffic.

   

  • Some endpoints in this category may not be hosted in Microsoft datacenters.
  • Some endpoints in this category may not have IPs provided for them (so require a proxy or other unrestricted path)

   

  The information discussed above is be available from the new Web Service and I have published a script which will obtain this information for you using the latest data. Microsoft official script information can be found here.

   

  Hopefully this new categorization helps your organisation to simplify the connectivity requirements for the Office 365 service by optimizing the core endpoints which really require it. Please leave your feedback for this feature at the Tech Community site where we have our official information available.

  Paul

In late 2018 Microsoft made some significant updates in the connectivity space for the Office 365 service. This consists of two elements, a new method to provide the URL & IP requirements for the service which will help our customers manage and automate the requirements, and secondly, as part of this new service, we've added an improved categorization of the URLs required. This second element is designed to help customers optimize traffic which really warrants special treatment, then allowing the rest to go via the standard egress model. This second element is outlined in more detail here.

In this post I'll cover this new method for the management of URLs & IPs for the service.

Previously, the place to go for connectivity requirements to Office 365 has been our URL & IP page which consisted of three elements:

1. The web page itself, for human review of the requirements broken down per service
2. The XML file which was used for device configuration
3. The RSS feed for change notification

We had a challenge with this solution however. As the Office 365 service grew, the number of URLs and IPs required to access the service has obviously increased in line with this growth, also the number of connection options has also increased with the addition of Expressroute. Customers have been reporting to us that they are running into administrative challenges keeping pace with the rate of change in this space and the old solution to provide this data was not meeting their needs as it often required manual management of the changes.

Another challenge was around how the URLs need to be handled, some have IPs provided, some don't, some live on Microsoft infrastructure, some don't, some are required, some aren't, some can use ExpressRoute for Office 365, some cannot. The XML file just did not provide the data in a way which allows customers to deal with these attributes as they require, nor automate or script based on this information.

We therefore worked to improve how we provide this information to meet our customers' needs moving forward which went GA in September 2018.

Data is now provided in JSON format via a Web Service and is tagged with various attributes to allow you to script and manage the data as you see fit, some of these attributes are new, such as the 'Optimize', 'Allow', 'Optional' tags plus Expressroute. (More on this in another blog post here)

Benefits for administrators from the Office 365 IP Address and URLs web service and the new categories include:

• Automation of Office 365 endpoint data and changes publishing
• System readable data for direct network device integration that is also script friendly
• Data available in JSON for scripts or CSV format for Excel
• Includes the new Optimize, Allow, Default categorization of Office 365 endpoints
• Includes ExpressRoute routable flag for each endpoint
• Version change notification published alongside the data
• All provided attributes are supported by owning development teams

Previously, manually created example PAC files were created and published every month, with the new system it is now possible to code this easily. Some example scripts can be obtained here.

The change cadence we aim for with regards Office 365 connectivity requirements is that we provide updates once a month, at the end of the month, with one month's notice and this cadence will continue with the new solution. There can be occasions however where

Paul

2. Localized network egress as close to the user as possible with matching DNS resolution

To enable your Office 365 traffic to take full advantage of Microsoft's global network, ideally, we'd want this traffic to egress locally and thus peer onto Microsoft's network as quickly and as close to the user as possible. However, backhauling to a centralized internet egress is common for an enterprise so as to facilitate consolidation of network pipes and security equipment such as firewalls and proxies. This policy is completely understandable for an organization with many sites to manage, and where the majority of business-critical traffic is kept internal to the corporate network. However, for SaaS services like Office 365, local egress becomes vitally important and as the enterprise starts to move to the cloud, it's likely also time to assess whether the current design for accessing the web, is still valid and optimal as we make this transition.

Using a centralized egress for all office locations in a single small/medium sized country is still a valid approach but we would want to ensure the latency to the egress is as low as possible, and if local egress is possible (even if just for Office 365 traffic) then this approach should be investigated for feasibility.

For example, Contoso's EMEA corporate HQ in London may host the internet egress for all UK sites, and as long as the connectivity to this egress is optimal, and we've optimized the connectivity from there to Microsoft's network, including peering, then performance should be excellent.

However, using this egress for the Contoso's EMEA sites in Athens, and Lisbon, may see the latency to the egress be higher than optimal and thus a local egress should be considered at these sites. The same goes for larger countries such as the USA where it would not be advisable to, as an example, backhaul traffic from east coast to west coast to egress.

Take the example below, Contoso has a branch office in Sydney Australia which uses the MPLS circuit to egress traffic in it's USA based head office. This traffic goes through the on-premises internet proxy and out to the internet in the USA via DLP devices and a firewall.

Whilst this model may work well for the pre-cloud world, where our services, data and applications sit in the head office site and for general web browsing, it allows Contoso to centralize its network egress and internet security stack. However, for distributed SAAS services and many other cloud services, this model no longer makes sense from a performance, efficiency, or cost perspective.

So, isn't it better than I backhaul this traffic to the head office on my managed network where I can egress it closer to the tenant location? In most cases no, this is not the optimal approach. Microsoft has a great deal of infrastructure globally which is designed to deliver the highest levels of performance for our services, therefore if we egress Office 365 traffic locally, we can take full advantage of this.

Firstly, we have Microsoft's global network.

Did you know Microsoft owns one of the largest networks in the world?  It's one of Microsoft' greatest unsung assets and it's part of Microsoft's multi-Billion Dollar investments to enable a truly global, dynamic cloud infrastructure. This network consists of very high bandwidth, low latency, failover capable links with 100k+ of route miles of privately-owned dark fiber. Multi Terabit connections from the Datacenters to each other, and the internet edge.

We peer with over 2700 ISPs globally in 190+ points of presence where we peer with ISPs to pull traffic from them onto our own network. This network is optimized to get your traffic to and from its destination as quickly as possible and covers all Microsoft traffic, from Azure to Office 365, to Xbox Live, to Bing. We can also deliver elements of some services from the edge of this network to improve performance (such as OWA) or provide network optimizations to connections to improve performance. You can read more about this network here and here.

The list of peer points (correct as of June 2019) can be found below and we often have multiple locations in these cities. For a full list and to see which other ISPs have presence in these locations, peeringdb is a public site where you can look this up, Microsoft's ASN number (network ID) is 8075.

Given the scale of this network, in most metropolitan areas of the globe your traffic should only be on the internet path for a very short period until your ISP hands it to Microsoft's network and we'll look at how you can assess this in the next section. By egressing traffic locally to the user, the traffic uses Microsoft's optimized network to backhaul to the data location for your Office 365 services, both saving money on private backhaul, and also higher performance.

In addition to this global network, Office 365 services have distributed service front doors around the globe in datacentres and also at the edge of the global network. All designed to deliver high performance to users regardless of where their tenant and data at rest resides. What does this mean? Let's look at the three main services which deliver this benefit in different ways.

Exchange Online

Exchange Online uses local frontend servers referred to as CAFE servers (which is an acronym for Client Access Front End) around the globe. When local egress is performed alongside local DNS resolution, clients will connect to a CAFE server close to where the user is regardless of where the Tenant resides, meaning low latency to the service 'front door'. Mailbox data is then backhauled between the CAFE and the active mailbox server over Microsoft's managed network.

SharePoint Online & OneDrive for Business

SharePoint Online works differently to Exchange Online by using an Anycast IP to find the nearest service front door which normally resides at the edge of the global network. This means SharePoint Online and OneDrive for Business traffic is connected through this distributed service front door where connectivity is done locally then optimized back to wherever the data at rest is. Again, the client should see low latency to the service front door in most scenarios.

Teams Media traffic

Teams media traffic has a complex set of connectivity possibilities depending on the scenario; however, it does have the ability to use a local transport relay server to bounce calls off in some scenarios. Imagine Contoso's tenant is USA based but im based in the Sydney office. I need to have a call with a colleague who is at a customer site in Melbourne. If we are both on different networks, then this call will have to go through a Microsoft relay server (as opposed to peer to peer when on the same network). If we both have local egress on our respective networks, this call can bounce of a local, Australian, Transport Relay server rather than one in the tenant location (USA).

So, by performing local egress we get to utilize local service infrastructure and networks for our cloud services, delivering performance and ultimately cost savings. Therefore, what does a modern, cloud first network look like for Contoso? Well one model we're seeing customers have great success with, is replacing the router at the branch site with an SDWAN device which not only has the MPLS connected, but also local internet circuits. This SDWAN device can then be configured to read the Microsoft Rest API for the Office 365 endpoints and selectively route the desired endpoints which we wish to go direct, such as the 'Optimize' marked endpoints via the internet path. Other endpoints such as Windows or Office updates can also be directed out locally in addition to any other known or trusted endpoints such as other cloud services.

This not only reduces load on the MPLS circuit, often by a significant amount, thus allowing Contoso to perhaps downsize it (and save costs) whilst maintaining performance over it, but it also greatly improves service performance by utilizing Microsoft's local infrastructure to deliver the service. We also see customers achieving great success in switching to using this egress to provide local internet access for the branch by utilizing cloud base secure web gateways (proxies) to provide general web browsing, which is a much more scalable and performant model for the modern world than backhauling to centralized on premises devices.

Local DNS resolution

A key element to this second element of the connectivity principles is local DNS resolution. This is because some Microsoft services utilize the location of the DNS resolution to provide local front doors to the respective service. For example, Exchange Online uses Anycast DNS to provide the nearest available CAFE servers to the client doing the lookup. If DNS is not performed near to the egress location then there is a risk the connectivity will be to a non-optimal CAFE server, potentially causing a performance impact. Utilizing a local DNS server also potentially reduces the latency for resolution, further improving performance. SDWAN solutions like the one outlined below should be able to assist in delivering this element of optimization.

In summary, getting your core Office 365 traffic out locally is the advised methodology which allows Microsoft to backhaul & optimize from a location close to the user. Some consolidation of egress locations makes logical and financial sense, but care should be taken to ensure unnecessary latency isn't added by having the egress too far away from the user location. Tools such as psping can be helpful in accurately measuring latency to the service front doors which we'll look at in the next section.

Next up Part 3: Optimize Route length & Avoid Network Hairpins

Back

Whilst our official guidance around Office 365 connectivity can be found at http://aka.ms/tune/ and also here it's worth discussing in detail what best practice looks like in this space.

An area I spend a great deal of time working on, is how to best connect to an Office 365 implementation from a corporate environment. It's an area I speak to customers about on a daily basis and the wealth of options can make it a highly complex area to get right. There isn't a one size fits all model either, what's right for one customer will be the wrong thing for another which makes generalizing advice hard and we need the details to make a correct recommendation.

There is also the issue that (in my view) there isn't enough information out there on Microsoft's global network infrastructure to allow customers to use this information to their advantage. This is something we've aim to redress in Ignite sessions and also here.

The other complexity revolves around the fact the cloud is a moving goal post for connectivity, its very nature means endpoints change regularly, applications may change their connectivity type as improvements are rolled out, new services are added, and so on. As such we need a network design which abstracts the organization from these changes, allowing the fluid nature of the cloud to be invisible to the end user whilst allowing the power of the services to be delivered optimally and enable the organization to consume service optimizations when they are rolled out.

What we need to do therefore is to drive for some standard connectivity principles, and by doing this we can achieve the 'north star' of optimal, flexible and abstract connectivity to our cloud services.

Our official principles can be found here, but I'll dig into them in more detail in this post.

So, what are these standard approach elements from a Microsoft standpoint?

1. Differentiate traffic
2. Egress Connections as close to the user as possible
3. Optimize Route length and avoid network hairpins
4. Assess Network Security

It's a simple list but important to know the detail on how to implement each stage to achieve success.

1. Differentiate Traffic

Identify and differentiate Office 365 traffic using Microsoft published endpoints data

As part of the improvements Microsoft announced in 2018 to the way the required URLs and IPs for Office 365 are published, we have also changed how the URLs are categorized to help customers deliver best practice in terms of connecting to the required endpoints for the service.

In the old system, URLs are marked "Required" and "Optional" with a very large list of URLs in each category, published via an XML file.

Traditionally Microsoft advised that customers optimize connectivity to Office 365 for the "required" endpoints. That often means, not using traditional proxy methods at the network edge due to the bottlenecks which commonly occur at this point. This is often due to the load Office 365 puts on these devices. In addition, the work often done at this layer such as SSL inspection for AV scanning or DLP which is often replicated in the application/service itself when it comes to Office 365 endpoints. Proxying traffic also often means that in most cases, Skype for Business or Teams uses TCP for voice/media calls as opposed to its preferred method of using UDP directly, with a very likely drop in performance and quality. More on the challenges with using traditional proxies for SaaS services like Office 365 can be found in another of my blog posts.

The difficulty with this categorization model for the URLs (Required & Optional) is that applying this advice has become increasingly difficult for customers as the Office 365 service has grown. The number of endpoints in the "Required" list is considerable and in addition, some do not reside on Microsoft infrastructure (eg CDNs) which we therefore do not provide IPs for. A customer may therefore understandably want to apply SSL inspection on some of these endpoints due to the number and differing nature of them. It's also hard to correlate which URLs have which security features (such as AV scanning/DLP) applicable to it within the service, and thus don't require it at the network edge. The net result is that often key Office 365 traffic ends up being sent via an unoptimized path and encounters performance issues as a result, or it is a large challenge to optimize the full list of endpoints.

We therefore set about a piece of work to analyze all the URLs for the service and identify those which really require optimization to enable customers to focus on the traffic which really requires special consideration, i.e. those which are either very high volume in terms of bandwidth/transactions/connection count and thus put heavy load on proxies, or those which are particularly latency sensitive and require the direct path with minimal interference, for example Skype/Teams voice & video traffic.

This piece of work showed that if we concentrate on a very small number of endpoints and apply the highest levels of optimization then we have an enormous amount of impact on our Office 365 performance. This small list however accounts for 75-90% of Office 365 Bandwidth, connection and transaction count. These core endpoints are those which will put massive load on traditional proxy infrastructure and/or require that optimized path a direct egress best provides.

With this small list in mind, we can now work around the problem with traditional egress models in a much more precise manner, by concentrating on optimizing the URLs which both cause the problem (in terms of load) and also are the biggest victims of it in terms of performance.

We have therefore categorized the URLs into the following three categories to help customers deal with the endpoints in specific ways according to the needs of the endpoint and the business.

• Optimize for a small number of endpoints that require low latency unimpeded connectivity which should bypass proxy servers, network SSL break and inspect devices, and network hairpins.
• Allow for a larger number of endpoints that benefit from low latency unimpeded connectivity. Although not expected to cause failures, we also recommend bypassing proxy servers, network SSL break and inspect devices, and network hairpins. Good connectivity to these endpoints is required for Office 365 to operate normally.
• Default for other Office 365 endpoints which can be directed to the default internet egress location for the company WAN.

Optimize Category

URLs in the Optimize category all have the following characteristics:

• Are Microsoft owned and managed endpoints hosted on Microsoft infrastructure.
• Have IPs provided
• Are routable via ExpressRoute (If the circuit is authorized for this type of traffic).
• Low rate of change to URLs/IPs compare to other two categories
• Expected to remain low in number of URLs
• Are High volume and/or latency sensitive

The Optimize list at the time of publishing is outlined below.

The above (optimize) endpoints will require access to the following IP address ranges:

• Exchange Online
• SharePoint Online
• Skype for business

For all customers, the advice is that, for the endpoints noted above, where possible:

• If proxies are in use, send the above URLs direct in the PAC file and allow a direct path through a firewall/NAT which has an allow list for the IPs and Ports listed.
• Ensure devices and paths handling this traffic can scale to handle the demand in terms of port requirements and bandwidth.
• Bypass or whitelist these URLs on network devices which do SSL decryption, DPI or content filtering, or any other interference of the traffic which could delay it.
• Look at applying the security elements your require for these endpoints, within the service and thus allow the traffic out of the network without hindrance.
• Egress these endpoints as close as possible to the users so the traffic can hit Microsoft's infrastructure locally.
• For remote users ensure these endpoints are sent direct via split tunnelling if VPN solutions are used
• Ensure that DNS resolution for these endpoints is done at the same location as the egress path

The advantage of the above list is we can send a customer's specific, and high-volume OneDrive/SharePoint traffic (tenantname.sharepoint.com & tenantname-my.sharepoint.com) direct, and apply DLP in the service if required, whilst sending any other tenant traffic (*.sharepoint.com) which may be required for sharing/collaboration, via an inspecting proxy.

Whilst most customers do not require it, if strict control of which tenants are accessible then tenant restrictions can be implemented to control which tenants can be accessed from within the corporate network.

By optimizing this list of endpoints, you can ensure that the traffic which is likely to put the most load on your network infrastructure, and/or is latency sensitive is treated in such a way that the service performs at its best, whilst removing the risk of overloading your standard network egress.

Allow Category

Access to the URLs marked as 'Allow' are required for Office 365 to function, however they are not as sensitive to network performance and latency as those marked as 'Optimize'. The amount of bandwidth these endpoints will consume is a small proportion of the whole figure and also connection counts will be lower than the optimize category.

Allow endpoints have the following characteristics:

• Are required for the service to function
• Are dedicated to Office 365 and hosted in Microsoft datacenters.
• Are not as highly sensitive to network performance as the optimize category
• Are not comparatively high bandwidth or connection count endpoints
• Higher rate of change than the optimize category
• Not all endpoints in this category will have IPs provided

Whilst these endpoints are required for the services to work and aren't as sensitive as the Optimize URLs, they still are critical to the running of the service. A heavy delay to connectivity to one of these endpoints could cause performance issues for users. As such it is recommended that these URLS are optimized as far as possible.

Recommended advice in an ideal situation is as follows:

• If possible, bypass or whitelist Allow endpoints on network devices and services that perform traffic interception, SSL decryption, deep packet inspection and content filtering. If it is not possible to bypass these URLs, ensure the work is as light touch as possible, and on optimized, well scaled devices which minimize any delay.
• If proxying this traffic, ensure proxy authentication for these URLs is disabled to remove any delay that may cause.
• If possible, prioritize the evaluation of these endpoints as fully trusted by your network infrastructure and perimeter systems.
• Egress these endpoints as close as possible to the users so the traffic can hit Microsoft's infrastructure locally.
• Ensure that DNS resolution for these endpoints is done at the same location as the egress path
• Prioritize these endpoints for SD-WAN integration for direct, minimal latency routing into the nearest Internet peering point of the Microsoft global network.

   

  Default Category

   

  This final category represents endpoints for Office 365 services and dependencies that do not require any optimization and can be treated by customer networks as normal Internet bound traffic.

   

  • Some endpoints in this category may not be hosted in Microsoft datacenters.
  • Some endpoints in this category may not have IPs provided for them (so require a proxy or other unrestricted path)
  • If these endpoints are not accessible, the services themselves will work generally, however certain elements will stop working. Eg help functions

   

  The information discussed above is be available from the new Web Service and I have published a script which will obtain this information for you using the latest data. Microsoft official script information can be found here.

  Next up – Part 2 Network Egress as close to the user as possible

However, my colleagues have run into an issue with multiple customers which causes mailbox migrations to Office 365 to be very slow or to fail completely which is very difficult to diagnose and as such I thought I'd share in the hope it helps some of you.

We've had numerous issues with mailbox migrations running slower than they should, network checks show good bandwidth, low latency, TCP Window Scaling enabled etc etc but still no joy. Errors such as "Transient error CommunicationErrorTransientException has occurred. The system will retry". Or some other timeout errors may also occur.

My colleagues Andres Canello and Roshan Padmanabhan have bottomed out this issue which is caused by a Denial of Service protection feature on many web egress devices such as proxies and firewalls called HTTP Flood Mitigation.

This security feature is designed to minimize multiple HTTP requests from the same source IP, during mailbox migration O365 will send many HTTP connection requests to the MRS endpoint as it goes about its business. Customers normally publish their MRS endpoint using a reverse proxy (for example TMG Server) and HTTP flood mitigation is enabled by default on TMG which is throttling the number of connections that can pass through it per minute, ultimately slowing or causing the migrations to fail.

For TMG you can disable this feature or, ideally add IP exceptions for O365 IP ranges to ensure this restriction is not applied for the mailbox migration traffic. However, given the enormity of our IP ranges for Office 365 it may be easier just to disable this for the duration of the migrations.

There are numerous settings which can affect this but the main one to be concerned with is HTTP requests per minute per IP Address.

Refer this link for more information on how to do this in TMG: http://community.office365.com/en-us/w/exchange/office-365-move-mailbox-fails-with-transient-exception.aspx

You can read more about Flood Mitigation here http://technet.microsoft.com/en-us/library/cc995196.aspx

It's also possible that you have similar capabilities on any other proxy or firewall device that's being used to publish your MRS endpoint so, even if you are not using TMG it's well worth checking if you have any DoS type capability on your proxy or firewall preventing multiple HTTP Requests from the same source IP.

This may not be called HTTP Flood Mitigation but will refer to HTTP or TCP throttling or DOS protection. QOS devices could also do something similar to limit the amount a single TCP session can use.

Hope this helps and Kudos to Andres & Roshan for their work bottoming this one out.

This can prove tricky with Office 365 as the information is almost always encrypted within an SSL session, however there are a number of methods we can use to look at how the application itself is performing and how long the requests we send to the remote server are taking to get a response. From this we can see if we've got a problem with the client or on the Datacenter side.

When we're working on-prem this is a lot easier. We can normally match up a request to a response as long as it's in the clear by using a network packet sniffer such as Netmon or WireShark, so we'll be able to see an RPC call to its response in Outlook and a HTTP GET to its response in SharePoint. When these requests are encrypted then that becomes impossible. So how do we do it? Well the methods vary with each product:

I've tried my best to make the images viewable without clicking out, any smaller ones should have a link to view full page if you need though.

Outlook

Outlook performance can be a tricky one at the best of times, especially so when using HTTPS.

We can however use the inbuilt connection status tool to look at the performance. Ctrl + Right click the Outlook icon in the bottom right of your task bar and click on 'Connection Status'. This will give you a whole heap of information on Outlook's connectivity.

Here you can see the Outlook 2013 output (the format and content varies with versions of Office)

From here I can see how many connection to Exchange I've got, the type, and some information on the RTT and processing time (Avg Resp & Avg Proc respectively). We can use these two values together to see the RTT as measured by Outlook. If we take the cached connection with 4963 requests then we have the following:

• Avg Resp: 29
• Avg Proc: 6
• Avg Resp shows us the RTT measured by Outlook.
• Avg Proc shows us the processing time, how long the RPC processing latency is, how long the server took to construct the response. If this is high it indicates a problem on the server side.

By subtracting 6 from 29 here, we get the latency which is 23ms

To confirm this I can use PSPING to connect to the mailbox and this shows an average of 20ms.

A great blog by a Microsoft colleague here describes these tests in more (and better) detail but this is a good test with inbuilt tools to show if we have latency or a delay on the Exchange side. The blog also outlines some great steps to take to look at other ways at the client and Outlook performance to see if there are any issues there

I normally also take a network trace whilst starting Outlook, then whilst performing actions such as opening a new mailbox, switching calendars, sending a large mail and then analysing the traffic for symptoms described in my blog post.

If you're running a newer version of Windows, you can also use Resource monitor to get a view on your round trip time for Outlook connections.

SharePoint:

There are a number of tools we can use to look at the page load performance within the browser.

If you're using IE then the inbuilt tools are a good starting point, especially with the newer versions of the browser.

By hitting F12 and using the inbuilt tools to trace the page load, we get information on each element of the page and how long it took to load and how big it is.

Here we see the URLS opened and the HTTP response code.

IE F12 Tools:

And over to the left of the same screen we get more information on how long it took for this to complete.

Fiddler:

Fiddler is a tool which inserts itself in front of the browser and allows us to capture encrypted requests in the clear and shows us the time it takes for each request to complete and allows us to spot any problem elements of the page which is slowing us up.

Here you can see information similar to that of the F12 tool but with more data.

Over to the right when we click on a URL we see detailing information on how long each stage of this connection tool. For example we have information on how long it took to get the server response, and to complete this response. This sort of information is enough to give us an indication on a slow loading SharePoint page how quickly we got the initial response and subsequently all the data.

If we select multiple URLs and click on Timeline we can see a graphical view of how long each stage took.

This is just intended as an introduction to the tool, the help file is pretty good as is the support community on the website, and there is a book if you're keen. As this is a third party tool (i.e. non Microsoft) I can't vouch for it but I know we use elements of the tool in our new Message Analyzer tool.

HTTPWatch

My personal tool of choice however is HTTPWatch. This is a free tool which works with most browsers but you'll have to buy the full version if you want the extended features but is well worth it if you do this on a regular basis and your boss will stump up for it!

This essentially acts as a proxy in front of your browser and allows it to see the elements of the page as they load and for me is the easiest to use and understand whilst giving me some great information on what the performance is like. Again, this is a third party tool and I can't vouch for it but we do use the full version within Microsoft and I personally use it extensively.

I'll use IE as the example browser here but the tool works with Firefox on Windows. Once installed, if you hit F4 to get the menu up, you should see HTTPwatch as an option, click that and a window should open up at the bottom of the tab.

Hit record then enter your URL. Here I opened my test SharePoint page in Office 365. You can see clearly in the timechart which section took the longest to load (the one highlighted took 0.9 seconds) and if you had a poorly performing section of the page, this would be as clear as day in this timechart. I've also hovered over the green line which indicated when the page's rendering started in IE. So I can clearly see here that at 1.4 seconds after entering the URL the page was visible to the user (although some elements were still coming in, in the background).

In addition to troubleshooting slow elements of a page, you can also can use this green line to measure a baseline of page load times, either for comparative purposes to an on-prem solution or perhaps before and after a network or page structure change. It's also useful to be able to compare page load performance from different sites.

If I then click on the time chart for that URL that took 0.9 seconds I can see where that time was spent in more detail. Here we can see the connection and SSL handshake took no time at all, but we spend most of our time waiting for a response from the server. Once we get the response, we receive the data in 0.1 seconds. In this example, 0.7 seconds waiting isn't too long a time but if this information gives us some great ideas on where the problem is. If receive was longer than expected, then perhaps we've got a slow network, or one of the other network tuning issues in my blogs is causing it to take a long time. Let's imagine Wait is the longest (like below) but taking 10 seconds. This would indicate to me that perhaps the SharePoint server is taking a long time to construct a response, have a look at the URL, what is it doing? Is it a poorly performing script or similar?

Alternatively this could have been caused by packet loss, perhaps the server didn't get my request for 9 seconds as we had to retransmit it? As we're using the professional edition we can see the local TCP port used for this connection in the columns and thus, we can (and I often do) take a simultaneous network trace and I can use this port information to isolate the TCP session that this GET request correlates to and look at the network performance. If there are retransmits, even with SSL they will be visible in the network trace, you just won't be able to see what call it was which was retransmitted.

So, there are an array of tools which enable us to troubleshoot and baseline the application layer, I've only scratched the surface of what the tools can do but hopefully it gives you at least a starting point to look at application layer performance with O365 (or on prem for that matter).

Paul

One of the main things we need to get right to ensure the most efficient and speedy connectivity to O365 is where in the world your DNS call is being completed. You'd think this wouldn't matter, you do a DNS lookup for your O365 tenant, get the address then connect right? Well, normally yes, but with O365, especially with Outlook, we do some pretty clever stuff to utilise our worldwide array of datacenters to ensure you get connected to your data as efficiently as possible.

Your Outlook connection will do a DNS lookup and we use the location of that lookup to connect you to your nearest Microsoft Datacenter. With Outlook we'll connect to a CAS server there and use our fast Datacenter to datacenter backbone network to connect you to the datacenter where your exchange servers (and data) are located. This generally works much quicker than a direct connection to the datacenter where your tenant is located due to the speed of the interconnecting networks we have.

http://technet.microsoft.com/en-us/library/dn741250.aspx outlines this in more detail but a diagram nicked from this post shows how this works for Outlook/Exchange connectivity when the Exchange mailbox is located in a NA datacenter but the user is physically located in EMEA. Therefore the DNS lookup is performed in EMEA, we connect to the nearest EMEA datacenter, which then routes the connection through to your mailbox over our backbone network, all in the background and your Outlook client knows nothing about this magic going on behind the scenes.

If your environment is making its DNS calls in a location on a different continent to where the user is physically located then you are going to get really bad performance with O365. Take an example where the user and Mailbox is located in EMEA. Your company uses DNS servers located in the USA for all calls, or the user is incorrectly set to use a proxy server in the USA, thus we're given the IP address of a USA based datacenter as that's where we think your user is located. The client will then connect to the USA based datacenter which will route the traffic to the EMEA datacenter which will then send the response back to the USA based datacenter which will then respond to the client back in EMEA. So with this scenario we've got several unnecessary trips across the pond with our data.

It is therefore vitally important to get the DNS lookup right for when you move to Outlook on Office 365.

So how do you check this? Well it could be a bit tricky as although we release a list of IP addresses used for O365, we don't tell you which ones map to where, for many reasons including the fact they change regularly. Thankfully one of my Microsoft colleagues has shown me an easy way to check you're connecting to a local datacenter.

All you need to do is open a command prompt on the client and ping outlook.office365.com and the response will tell you where the datacenter is you'll connect to. So sat here in the UK at home, I get EMEAWEST

If I connect to our Singapore VPN endpoint and turn off split tunnelling and force the DNS call down the VPN link (our Internal IT do a great job of making these things configurable for us techies) then I get directed to apacsouth.

And if I connect via VPN to the mothership in Seattle, my DNS call is completed there and thus I get directed to namnorthwest.

So it's a quick and easy check, just make sure the datacenter returned is in the same region as you're physically located in.

SharePoint is currently directed to the datacenter where your tenant is located so it doesn't matter so much where the call is made for this (although it should still preferably be local to the user for the portal connection). Lync is slightly different and is outlined in this article in more detail.

It's also worth ensuring all your clients are using a proxy in the same region as where they are located, as if not, they could hit the problem outlined above and thus be getting unnecessarily poor O365 performance.

Maximum Segment size or MSS is a TCP level value which is the largest segment which can be sent on the link minus the headers. To obtain this value take the IP level Maximum Transmission Unit (MTU) and subtract the IP and TCP header size.

So for a standard Ethernet connection with minimum size IP and TCP headers we subtract 40 bytes from the 1500 byte standard packet size (minus the Ethernet Header) leaving us with an MSS of 1460 bytes for data transmission.

So to get the most efficient use of a standard Ethernet connection we want to see an MSS of 1460 bytes being used on our TCP sessions.

This setting is agreed in the TCP 3-way handshake when a TCP session is set up. Both sides send an MSS value and the lower of the two is used for the connection.

It's easy to check this, take a Netmon or Wireshark trace and find the connection you're interested in, Netmon will filter the connections by process on the left hand side for you.

Once you've found the connection (ensuring you've started tracing before initiating the connection) then you just need to open the first to frames of the connection, the SYN & SYN ACK. Indicated by an S followed by an A..S in the description of the frame. To capture the 3-way handshake make sure you start tracing, then start Outlook, or connect to your SharePoint site in a new Browser window.

Once you've clicked on the first packet, the SYN, then in the frame details down on the bottom, open up TCP Options and the MSS can be clearly seen.

Here we see the MaxSegmentSize shown as 1460.

Repeat this with the SYN ACK which should be the second frame if you've filtered the connection away from other traffic. The lower of the two values will be your MSS. If it's 1460 then you're configured to use a full sized data payload.

One caveat to this, it doesn't mean that this value can actually be used, it's possible a network segment along the route has a lower MTU than we're aware of. If this is the case, if all is well we'll get an ICMP message back from the router at the edge of this link when we send a 1460 byte packet with the do not fragment bit set. This packet will tell us what the MTU is on the link and we'll adjust accordingly. However it's always worth checking this value is set to a high value and we can see the TCP payload throughout the trace is at 1460 (on full packets) and hasn't dropped down to a lower value.

It's common to see this value lower than the maximum of 1460 (for an Ethernet network), if for example we know a network segment along the route has a lower MTU, one with an encryption overhead for example, but the value shouldn't be significantly lower. 576 Byte packets are a sure sign we've hit problems and dropped down to the minimum packet size so keep an eye out for those.

Also, remember, if you're using a proxy, you'll have to check this both on the client, and a trace on the proxy or NAT device if used as there will be two distinct TCP sessions in use and you won't see the problem if it is beyond the proxy/NAT unless you trace there for that second TCP connection.

It's rare to see an issue with this, but it's always worth a quick check to ensure it's working as expected.