REEF Graduates to a Top-Level Apache Project

This post is authored by Markus Weimer, Principal Scientist, Beysim Sezgin, Principal Engineer, and Hiren Patel, Program Manager, all at Microsoft.

Raghu Ramakrishnan, Microsoft’s Chief Technology Officer for Data, recently shared some behind-the-scenes details of Azure Data Lake. We are excited to share some important news regarding one of those systems today:

In November this year, Apache REEF became the latest top level software project at the Apache Software Foundation. Graduation to a top-level Apache project signifies a major milestone for the project, as it is now owned and governed by a diverse community.

REEF started at Microsoft to facilitate the rapid development of Big Data applications on Apache Hadoop YARN in any combination of JVM and .NET languages. REEF became Apache-licensed Open Source Software in 2013 and entered the Apache Foundation’s incubator in late 2014. During incubation, the community steadily grew in contributors, committers and users. Today, REEF forms the backbone of production services such as Azure Stream Analytics and attracts contributions from other teams both from Microsoft and outside. Further, REEF enables research at several top-tier universities, including at University of Washington, TU Berlin and Seoul National University.

With Microsoft’s continuing focus on openness including Making Azure HDInsight available on Linux and our contributions to Apache Hadoop, the graduation of an Apache project is an important milestone for us. It symbolizes the changing relationship between Microsoft and the Open Source community. A huge “thank you” to the REEF community within and outside Microsoft, and to our mentors who have supported us and been steadfast through this process.

Looking forward, REEF is enabling the next wave of Big Data applications on Azure Data Lake. Expect us to share more on that in the future. And, as always, contributions are welcome!

Markus, Beysim & Hiren
Follow Markus on twitter or at his website.