DAVID CROWLEY, Managing Director, Network Enablement

I’m at International Telecoms Week (ITW) today, talking about how the telecoms industry is in the midst of a change to accommodate the growing trend of data consumption. As we look at how people and businesses will interact with technology in the future, there are investments we need to make now to support our customers and help their businesses grow. As people and organizations expect data and information at their fingertips Microsoft must have an infrastructure that can deliver the cloud services, including Azure, which our customers need to support their global businesses.

Over the past 9 months, Microsoft has been significantly investing in subsea and terrestrial dark fiber capacity by engaging in fiber partnerships that span multiple oceans and continents. And today, our connections across the Atlantic and Pacific just got stronger.

We announced deals with Hibernia and Aqua Comms, in which Microsoft is investing in a cable with each company to connect Microsoft’s datacenter infrastructure from North America to Ireland and on to the United Kingdom. These cables will help deliver data at higher speeds, with higher capacity and lower latency for our customers across the globe.

Additionally, we joined a consortium comprised of China Mobile, China Telecom, China Unicom, Chunghwa Telecom, KT Corporation with TE SubCom as the cable supplier. As part of our participation in the consortium, Microsoft will invest in its first physical landing station in the US connecting North America to Asia. The New Cross Pacific (NCP) Cable Network will provide faster data connections for customers, aid Microsoft in competing on cloud costs, all while creating jobs and spurring local economies. The goal of our expansions and investments in subsea cables is so our customers have the greatest access to scale and highly available data, anywhere.

Satya has discussed his vision of Microsoft being a mobile-first, cloud-first company. During Microsoft’s latest earnings we shared that the Commercial Cloud division, which includes Azure, Office 365 and other services, grew 106 percent, and as we expand our cloud services and global infrastructure, we need a strong subsea strategy to ensure our customers experience high availability access to their data.

When we look to the future with these investments, we believe our customers will see that Microsoft is pulling together all the components necessary to make its cloud services the most reliable, accessible and secure.

Competition in the cloud and infrastructure space continues to heat up. But it’s not a battle that will be won on just cloud or infrastructure alone, but instead on holistic innovation and providing value to customers from the “sea to the sky”.

For more information on our involvement with Aqua Comms, Hibernia and NCP check out their press releases.

Brad Booth, Principal Service Engineer, Networking

On April 15, 2015, I am delivering a keynote address titled “From Silicon to System: How Hyper-scale Impacts Your Design” at the Ethernet Technology Summit being held at the Santa Clara Convention Center.

Microsoft’s network is the second largest in the world. Even with such a large footprint in the Ethernet industry, many vendors and suppliers can hardly fathom the size of our network. It’s bigger than Enterprise, bigger than mega-scale… to the point that Microsoft coined a new term: hyper-scale. At this scale, a change that may seem insignificant when you look at one switch or one network interface card gets magnified by the million or more devices we have in our networks. Talk about an excellent example of the butterfly effect.

In my keynote, I plan to talk about technologies that Microsoft is involved in: hierarchical SDN (HSDN), flexible Ethernet (FlexE) and consortium for on-board optics (COBO).

HSDN is an effort that Microsoft initiated in the Internet Engineering Task Force (IETF) to get past the issues being faced with the size of forwarding tables required in switches.

FlexE is a project in the Optical Internetworking Forum (OIF) to permit Ethernet devices to scale with better granularity to the network requirements. For example, if a link between datacenters is only capable of 75 Gb/s throughput, it is better to operate the link at that rate rather than a standardized sub-rate.

COBO is a Microsoft initiated effort to move the optical modules used in our network equipment away from the faceplate to inside the system. This will help reduce power while improving the airflow and efficiency of the network equipment as we continue to scale bandwidth beyond 100 Gigabit Ethernet.  

The Microsoft network team has been engaging the industry to help them understand the butterfly effect when operating a hyper-scale network. We hope these efforts will help the Microsoft Cloud better serve our customer’s needs, while providing an agile, scalable, reliable, visible, and efficient network. I hope I will see you at the Ethernet Technology Summit.

Guest post by Shaun Harris, Principal Hardware Engineer

Microsoft has been designing, building and operating datacenters for over two decades and this experience has led to several key insights into driving efficiency at all levels of the datacenter design. This is why we chose to join the Open Compute Project (OCP) just over a year ago – to share our learnings with the community. Since joining, we have delivered the Open CloudServer (OCS) specification to the project, in addition to numerous other innovations.

Today at the Open Compute Project (OCP) U.S. Summit, Microsoft continued its innovation with the announcement of the contribution of a novel distributed Uninterrupted Power Supply (UPS) technology, which we call Local Energy Storage (LES). We are excited to make this hardware innovation available to the OCP community so that its benefits can be realized by a broader set of IT customers.

LES offers an integrated power supply and battery combination which is fully compatible with the Open CloudServer (OCS) v2 chassis system. The new LES units are mechanically interchangeable with the previous PSUs (common slot compliant) that were announced at the time of OCS v2 launch at the OCP EU Summit in Paris last year. The IT administrator can now choose which PSU to use for a specific type of deployment, depending on existing datacenter topology and battery backup requirements.

Let’s take a deeper look at the challenges behind traditional UPS designs, the key technological innovation behind LES, and the advantages of deploying LES as a distributed power backup solution.

Challenges with traditional facility UPS power backup solutions

In a traditional datacenter environment, the UPS system is housed in large separate rooms and typically consists of a large array of flooded lead-acid batteries stringed together to provide the interim backup power for the IT load in case of utility power outages (until the generators are turned on). The graphic shows this type of traditional design with the dual redundant facility UPS. This type of traditional UPS design leads to several inefficiencies for the datacenter deployment:

• The UPS and battery equipment room footprint accounts for 150,000 ft² or approximately 25 percent of the total facility footprint (for a typical 25MW datacenter). Assuming a rate of $220/ft², the UPS room construction and buildout accounts for $31 million in datacenter capex spend.
• The power stages upstream of the IT load are comprised of AC/AC or AC/DC conversions before the delivering input voltage to the server. This type of double conversion UPS design leads to inefficiency in the power backup system which negatively impacts the Datacenter PUE by up to 17 percent. This is comprised of 8 percent set aside for battery recharge, 1 percent internal systems management and 9 percent for losses in double conversion.
• Traditional UPS facility systems have a large failure domain that impacts availability for the entire IT load that is supported by the power backup solution. In case of UPS failures, this can lead to unexpected service outages that impact hardware availability and customer SLAs. To mitigate against this risk, datacenter designers add reserve UPS systems, coordinated power switchover systems, and safety margin to the UPS and battery systems. This creates a complicated solution for ensuring high availability power backup, which increases both the solution level capex and opex costs.

Leveraging our learnings building and managing datacenters, for the mechanical/cooling architecture, we moved to an Adiabatic based design which provided significant cost savings for the datacenter build and simplified the operational aspects. The graphic below shows the modified datacenter with the Adiabatic system, but still having the dual-redundant UPS design.

As we continued to evolve our datacenter designs for improved efficiency, we started to take a look at the power system distribution, so that we could address the various challenges highlighted above with the traditional UPS topology. The result of this research was a radical simplification to how the power solution could be designed. The key insight we had was that by completely eliminating the facility UPS and moving that capability directly into the IT load, we could not only achieve cost reductions and operational simplicity, but could open up a whole new area of architectural innovation by tighter integration of the battery system with the IT management and controls system. These observations led to the design innovations for LES, as outlined in the next section.

Key technology innovation in LES

Every server using a modern switch mode power supply unit (PSU) comes with active power factor correction, capacitive bulk storage, and an isolated DC/DC output stage. The DC/DC output stage pulls its energy from the bulk storage and maintains the proper output voltage tolerances across the blade load profile.  The LES topology reuses the PSU design and control loops (ensuring maximum leverage of already proven designs) adding only components such as batteries, battery management controller, low current isolated charger and a low current 380VDC isolated output.

The hand tool and electric vehicle industry has created a market for low cost, high performance, high quality Li-Ion cells. These same cells are used in the LES battery pack, to ensure that the LES solution can leverage industry volume economics and supply chain.

The LES design innovation takes commodity energy storage devices (batteries) and an industry proven PSU power train, fusing these together in a single package to maximize energy delivery efficiency and minimize cost overheads. LES integrates the battery in the 380VDC bulk capacitance section of the PSU.

Advantages of LES for Datacenter deployments

The simplicity of the LES design coupled with its cost structure and service model provides significant benefits for datacenter deployments.

• Up to 5x cost reduction over traditional facility UPS, achieved by extreme simplification of the datacenter power delivery solution and moving the energy storage function to a high volume commodity supply chain.
• Moving the energy storage local to the server eliminates up to 9 percent of the losses associated conventional UPS systems. The LES topology and lithium-ion batteries requires only 2 percent charge overhead versus conventional UPS systems (which require up to 8 percent charge overhead and 1 percent operating overhead). The net result is up to 15 percent improvement in Datacenter PUE
• Given no requirement for a UPS or battery room, the facility footprint can be reduced by 25 percent for Datacenter build capex savings
• Significantly improved serviceability model when compared to flooded lead-acid batteries in traditional UPS solution. LES units are hot swappable and safe upon removal, without any dangers associated with exposure to high voltage or chemicals. Distributing the battery in small power blocks delivers a ‘fail small’ system minimizing potential failure impact zones.
• By locally integrating the energy storage devices we have enabled low latency detection and controls not possible in a conventional centralized UPS systems. The LES unit when tightly coupled with the IT management system (OCS chassis manager) can enable new architectural scenarios for utilization efficiency – such as peak shaving, trough charge, processor state control from the row distribution to server. We will release more details on these opportunities in the near future.

We are excited to share the innovative LES design with the OCP community, and will be making available all design collateral for LES as part extending the Open CloudServer (OCS) specification. If you’re at the summit, please stop by our booth (B2) to learn more about OCS and LES.

Shaun Harris
Director of Engineering, Cloud Server Infrastructure, Microsoft

Read more: Microsoft Furthers Open Networking Specification

Two years ago, we announced a partnership with a number of private and public sector partners to build the first zero-carbon datacenter, called a Data Plant—a fuel cell-powered datacenter that simplifies the power distribution infrastructure by bringing  together the power plant with the datacenter to radically improve efficiency. We are excited to announce that yesterday, the new Data Plant officially opened with a “cable-cutting” ceremony at the water treatment facility that will be used to power the Data Plant.

This opening is a milestone in our ongoing efforts to pursue carbon neutrality through innovative, sustainable energy investments. “Our objective is to transform the energy supply chain of our datacenters toward greater efficiency and reduced environmental impact,” said Christian Belady, General Manager of Datacenter Services, Microsoft. “This is an important investment as Microsoft continues to pursue energy efficiency and clean energy projects. We’re excited that our private and public sector partners like those in Wyoming share our commitment to exploring innovative renewable solutions that can help lead to a more sustainable future for everyone.”

To make this concept a reality, Microsoft partnered with the University of Wyoming, Business Council, Cheyenne LEADS, Cheyenne Board of Public Utilities, the Western Research Institute, Connecticut-based FuelCell Energy, and other industry and state and local government partners. The project uses biogas produced by recycled common waste byproducts at the Dry Creek wastewater facility to power the fuel cell at the Data Plant. The fuel cell converts the biogas into electricity to power the Microsoft IT server container. The Data Plant is expected to produce 250 kilowatts of renewable power but only use about 100 kW, so the remaining power will be delivered back to the wastewater treatment plant to reduce its expenses by using the power.

Microsoft has made great strides to make datacenters more energy efficient. We see this Data Plant in Cheyenne as an important step in demonstrating a new way to power datacenters, and will also use the plant as a way to continue biogas and fuel cell research so that we can continue to innovate and find new ways to reduce our environmental impact.

Kushagra Vaid
General Manager
Server Engineering

Yesterday I gave a keynote address at the Open Compute Project (OCP) European Summit, where I announced the contribution of the 2^nd-generation Open CloudServer (OCS v2) specification.  In January, Microsoft joined OCP and submitted the first OCS specification, born from our experience in our own cloud-scale datacenters.  OCS v2 incorporates a number of improvements, delivering greater performance and more flexibility.  In addition, Microsoft Open Tech announced today the release of an open source reference implementation of operational management tools for OCS v2.  In all, we at Microsoft are very excited about the tremendous momentum we see in the Open Compute Project, and see it as an excellent community to foster more efficient datacenters and accelerate the adoption of cloud computing.

It’s been great to see tremendous support from industry partners contributing to the OCS v2 design.  At the summit, we will show an array of OCS v2 designs and platforms, including from the following partners: 

• Quanta QCT
• Wiwynn
• ZT Systems
• Hyve Solutions

In addition, a growing ecosystem of component vendors are expressing their support for OCS v2, including Intel, Mellanox, Seagate, Geist, Delta, and more.

Overall, the OCS v2 specification is the convergence of a number of design points.  Meeting these design points in one server is critical for Microsoft to meet its service delivery and datacenter operations goals, and also the needs of the community who’ll use OCS v2 for their own datacenter operations. 

The first design point for OCS v2 is to support a diverse range of cloud services.  The specifications we’re contributing to OCP reflect our long history in datacenter architecture and cloud computing.  To date, Microsoft has invested more than $15 billion in our global cloud infrastructure, and today we provide upwards of 200 cloud services to 1 billion customers and 20 million businesses in more than 90 markets around the world.  Over the last six months, the OCS v2 design has been thoroughly tested in our own datacenters, from powering IaaS and PaaS services in Windows Azure, to hosting e-mail and collaboration services in Office 365, to hosting latency-sensitive gaming services in Xbox Live.  Converging onto a unified, flexible design allows us to optimize the economics of our supply chain, while delivering a diverse array of cloud services from one underlying server platform.

Our second design point is ensuring OCS v2 can be deployed into any region or colocation facility around the world.  OCS v2 takes into account the variation of requirements across geographies, making it simple and efficient to deploy the design in a single region, or as part of a global footprint.  OCS v2 enables compliance with differing electrical standards that affect power and voltage to the rack, as well as safety and regulatory certifications that impact weight, rack size, mechanical, and cooling parameters. 

The last design point is to optimize performance relative to TCO.  OCS v2 delivers the high performance our online services need, while minimizing overall TCO.  Reaching the sweet spot for a number of server performance metrics was critical for OCS v2 to be deployed widely across Microsoft’s own global datacenter footprint.  OCS v2 introduces numerous new innovations, including: 

• A dual-processor design, built on Intel Xeon E5-2600 v3 processors, enabling 28 cores of compute power per blade, and reflecting Microsoft’s joint engineering collaboration with Intel to develop the next generation board.
• Advanced networking for low latency, high bandwidth, highly-virtualized environments, based on 40-gigabit Ethernet networking, with support for routable RDMA over Converged Ethernet (ROCEv2).
• Flexibility incorporated into the core design itself.  This allows the integration of a variety of components and add-on cards, including FPGA accelerators, which enables customers to tune their servers for their own unique workloads.
• Low-cost, high-bandwidth, Flash-based memory support, incorporating the latest form factor for m.2 Flash memory.  This allows OCS v2-based servers to incorporate higher-capacity SSDs, while ensuring TCO optimization by virtue of using cost-optimized NAND.
• A compact, high-capacity power supply, capable of delivering 1600 watts of power, with a high holdup time of 20 milliseconds.
• Support for high memory configurations, along with flexibility in the amount of memory deployed, by virtue of support for 128GB, 192GB, and 256GB memory capacity configurations.

Overall, we’re very excited about the industry enthusiasm for OCP, and how the organization is focusing the efforts of all the partners in our industry, to accelerate the rate at which innovations in our industry come to market.  We look forward to working with the community to help push datacenter innovation forward.  You can find more information about OCS v2 at www.opencompute.org.

//KV

Rob Bernard

Microsoft is committed to reducing our environmental footprint, and over the past two years we continue to meet our goal of becoming carbon neutral. Our approach to meeting that goal, however, continues to evolve. Today, we are announcing another move to make our operations more environmentally sustainable by signing a 20-year power purchase agreement (PPA) for wind energy in Illinois that will be funded in part by proceeds from Microsoft's carbon fee.

This is our second PPA, following the purchase in November of wind energy from the Keechi Wind Project in Texas. This most recent power purchase, the Pilot Hill Wind Project, is our largest wind investment to date. It is nearly 60 percent larger than Keechi, at 175 MW versus Keechi's 110 MW. This project builds on our commitment to renewable energy and our strategic objective to transform the energy supply chain toward radically greater efficiency and reduced environmental impact.

The agreement with EDF Renewable Energywill see Microsoft purchase up to an estimated 675,000 MWh of renewable energy from Pilot Hill Wind Project each year, enough to meet the power needs of 70,000 homes in Illinois. The Pilot Hill Wind Project is a 175 MW wind facility 60 miles from Chicago, IL, spanning the border of Kankakee and Iroquois Counties and the wind farm is on the same electric grid that powers our datacenter in Chicago. Pilot Hill Wind Project construction has already commenced and will begin delivering green power in 2015.

Because this is a new project, the energy generated at Pilot Hill is "additional," which means that our purchase is bringing new renewable energy onto the Illinois electric grid. By purchasing wind, we will reduce the overall amount of emissions associated with operating Microsoft facilities and hopefully spur additional investment in renewable energy. Because the Chicago datacenter draws power from the Illinois power grid, projects like Pilot Hill help provide a non-polluting source of energy that displaces greenhouse gas emissions from conventional power.

Microsoft's commitment to green power is now included in our Global Public Policy Agenda, and extends to our datacenters and our offices. Our datacenter in San Antonio, Texas uses recycled waste water for cooling and our Quincy, Washington facility uses hydropower as its primary source of energy. Last fall we announced a new proof-of-concept datacenter to bring the power plant inside the datacenter by mounting fuel cells onto the datacenter's server racks, and these fuel cell datacenters have taken one step closer to becoming a reality following a successful demonstration of the concept this year. In addition, over the past fiscal year, we have purchased more than 3 billion kilowatt-hours (kWh) of green power, equivalent to 100 percent of our global electricity use

We know that we still have work to do, and we will continue to pursue energy efficiency and clean energy projects, from smarter buildings to more efficient datacenters. The Pilot Hill Wind Project is another major step to continue our drive to reduce our environmental footprint and to be carbon neutral.

Check out our new infographic that describes how Microsoft is buying green energy.

Wind Turbine photo credits:  Sophie Brandstrom and Dennis Schwartz

Josh Henretig, Director, World Wide Environmental Sustainability

Microsoft is once again partnering with others to further fuel cell innovations, with the goal of developing a very efficient, low-cost fuel cell system. Microsoft, Redox Power Systems LLC, the University of Maryland and Trans-Tech Inc. were recently awarded $5 million by the Advanced Research Projects Agency – Energy (ARPA-E) to develop transformational fuel cells.

This research is being funded through the Reliable Electricity Based on Electrochemical Systems (REBELS) program, which seeks to encourage innovation around distributed stationary power generation by introducing technology concepts that have the potential for significantly lower costs and higher performance than existing distributed generation technologies. Though fuel cell technologies have long been recognized for high conversion efficiencies and the potential for near-zero greenhouse gas emissions, their high cost has been a major barrier to widespread adoption.

With this project, Microsoft will use its server racks to integrate and perform independent live testing on the enhanced fuel cells being created by the other companies and university. The goal is ultimately to create a system that would be significantly less expensive and more sustainable than traditional datacenter designs. “Our vision is to bring the power plant directly into the datacenter by integrating fuel cell stacks into every server cabinet, effectively eliminating energy loss that otherwise occurs in the energy supply chain and doubling the efficiency of traditional datacenters,” said Sean James, Senior Research Program Manager for Microsoft Global Foundation Services.

As I’ve written before, when it comes to environmental sustainability, Microsoft believes strongly in the importance of partnerships to help drive innovation. One partnership with the city of Cheyenne, the University of Wyoming and Fuel Cell Energy aims to build the first zero carbon datacenter, designed to run on biogas from landfills or water treatment plants. Meanwhile, a proof-of-concept datacenter that would mount fuel cells directly onto the datacenter’s server racks was successfully demonstrated at the National Fuel Cell Research Center at the University of California, Irvine. Most recently, we announced a three-year agreement with the University of Texas at San Antonio to research and develop distributed generation technologies that will transform how datacenters consume energy.

We look forward to sharing more progress in the future, and be sure to check back on the Microsoft Green blog for more on what else Microsoft is doing transform the energy supply chain. You can read the University of Maryland’s press release here.