Earth Day is April 22, and next year (2020) will be the 50th anniversary!  Leading up to that day, I’d love to share some videos which highlight the amazing work that our AI for Earth grant recipients are doing, using machine learning to solve hard challenges in agriculture, water, climate change, and biodiversity. 

Wild Me

https://www.youtube.com/watch?v=rQqao37u1wU

Wild Me is using computer vision and deep learning algorithms to power Wildbook, a platform that can identify individual animals within a species.  They also augment their data with an intelligent agent that can mine social media. 

SilviaTerra

https://www.youtube.com/watch?v=stNlw_CFklc 

SilviaTerra is empowering conservationists, government organizations, and land owners to inventory forests for ecological, social, and economic health using AI.

Tohoku University

https://www.youtube.com/watch?v=-kUvcLZpgRI

On March 11, 2011, one of the most powerful earthquakes ever recorded shook the northeastern coast of Japan, causing a tsunami that killed nearly 20,000 people. More than 2,000 remain missing. The Tohoku region outside of Tokyo was hit especially hard, forcing the local community to completely rethink how it responds to future disasters. With the help of a grant from Microsoft’s AI for Earth program, researchers like Bai Yanbing of Tohoku University are attempting to harness the power of artificial intelligence solutions to improve mapping technology so that relief workers might stand a better chance of saving more lives when the next earthquake hits. “Japan is very much earthquake country,” warns Professor Jun Murai of Keio University. “So the resiliency and readiness for the next one is very much a part of our life. The new challenge is to utilize the data and then to understand the way to recover from the disaster.”

SOS Mata Atlantica

https://www.youtube.com/watch?v=RbK5W99JipA

An estimated 35 million Brazilians don’t have access to clean water.  SOS Mata Atlantica plans to leverage Microsoft’s artificial intelligence technology to improve its clean water initiative. Nearly 3,500 volunteers currently monitor more than 250 rivers throughout the Atlantic Forest, which includes the vast majority of the country’s urban population and its biggest cities. Now, the NGO hopes to evolve from issuing annual reports to sharing more actionable insights – backed by strong data – that could impact public policy and save a precious resource from the ravages of pollution. 

If these stories inspire you, why not take part in the AI for Good challenge?  It’s open for submissions until April 24, 2019.  Take on big issues in agriculture, water, climate, biodiversity, and empowering people with disabilities.  You can also win Azure credits, a Surface book, Nvidia GPU, or Xbox One!  I’m looking forward to seeing what people build. 

.NET Rocks!

It’s always fun to speak with Richard and Carl on the .NET Rocks podcast. 

Can Artificial Intelligence help protect the Earth? Jennifer Marsman says yes! Carl and Richard talk to Jennifer about her new role working with Microsoft Research on implementing machine learning systems to support agriculture, environmental protection, biodiversity and more. Some of the projects originate within Microsoft, but there are a number of external projects supported by the team with Azure credits and funding. AI for good!

https://www.dotnetrocks.com/?show=1572

https://player.fm/series/net-rocks-134848/ai-for-earth-with-jennifer-marsman

TechForward

I also spoke with Cheryl Chotrani on the Tech Forward podcast. 

We talked about the AI for Earth program, FarmBeats, Project Premonition, bias in machine learning, and technology’s diversity issues. 

By running a global grant program, developing their own APIs, and funding projects, AI for Earth is, in Jennifer’s words, “a chance to use my passion for machine learning to make a difference.”  One such project is FarmBeats, which is interested in sustainable, long-term solutions to maximize yield and, ultimately, reduce world hunger.  Another initiative, Project Premonition, puts mosquitoes to work in order to predict outbreaks of diseases.  While machine learning is obviously a powerful tool, there is always the danger of bias.  Finally, when it comes to tech’s diversity issues, Jennifer identified two distinct, but related issues: attracting a diverse workforce, and maintaining that diversity.

https://goodbyteventures.com/episode-26/

The topic of my talk was innovations in data collection for wildlife conservation.  Data is the heart of machine learning, and there are some aspects of data collection for wildlife conservation that are particularly harrowing, such as current methods of tagging harming animals and poachers using data from wildlife scientists to target and kill rare species.  However, all is not lost – I have seen some interesting innovations in the area of wildlife data collection, many of them used by our AI for Earth grant recipients.  I spoke about these 5 areas of data innovation:

• UAV/drone imagery – projects such as FarmBeats have created optimizations for drones to collect data, like an autonomous iPhone app that allows a farmer or biologist to specify a path or area to cover and the algorithm calculates the best path to minimize battery usage and leverages the wind to speed up/slow down.  It also introduced the the innovation of leveraging television whitespaces (unused television channels) for connectivity; television wavelengths are much lower frequency than traditional WiFi and stretch much farther, so one TV Whitespaces router could provide connectivity to a large farm. 
• Camera trap – researchers at the California Institute of Technology (Caltech) have compiled a custom dataset of camera trap images to help develop computer vision models for identifying animals. Camera traps produce massive amounts of raw image data that must be sorted to find the useful photos, a task that would be ideal for automated computer vision systems. The Caltech dataset helps computer vision researchers refine their models to deal with previously-unseen locations by generalizing about backgrounds. When the models can learn to distinguish what’s typically in the background, they can ignore it and correctly pick out photos with the desired animals—leading to an efficient and scalable method to take advantage of all that camera trap data.
• Simulation – one difficulty in training computer vision models is providing the models with good data. Someone first has to go through the time-consuming manual process of assessing thousands and thousands of images to produce a dataset that will be effective for the machine learning process—as Caltech did for their camera trap project. However, advanced 3D computer modeling now offers a different possibility: creating simulated images of real-world animal habitats. Simulating images with tools such as AirSim would allow us to quickly build custom datasets covering the wide variety of conditions and situations the computer vision model might encounter, and train the models faster, moving on sooner to processing real imagery and getting useful results.
• Crowdsourcing – applications like iNaturalist allow citizen scientists (everyday people who care about the environment) to take pictures of wildlife (both flora and fauna) from their mobile phone and submit these observations to a database for biologists to consume.  The pictures can be easily augmented with date and location metadata from your phone, and a machine learning classifier can help non-experts identify the species by suggesting the top matches for the photographed species based on computer vision.  Therefore, the work of cataloguing the local wildlife is crowdsourced, and the computer vision helps people label their data properly.  
• Social media – the organization Wild Me created a platform called Wildbook which powers websites such as http://whaleshark.org.  This site uses social media for its success.  At 10pm every night, an intelligent agent searches various social media sites such as YouTube for pictures and videos of whale sharks.  When it finds instances of these animals online from people posting pictures or videos of their whale-watching trips, it can identify the animal to the individual animal level—not just recognizing that it is a whale shark in general, but which specific one it is.  This can help with tracking migration paths of individual animals and estimating population densities.  It also posts a comment on the YouTube video letting the poster know that their image was used for conservation purposes and linking to an informational page on the animal which contains relevant information on the animal, locations and times it has been sighted, and other animals with which it’s been seen. 

The AIWC organizers also released a challenge problem on determining an optimal flight path for a drone to fly through a simulated African environment to collect data on elephant location density.  Submissions for the challenge are still being accepted.   

The full list of accepted papers is at https://sites.google.com/a/usc.edu/aiwc/accepted-papers.  Check them out; there is some compelling work being done using machine learning to address hard problems in wildlife conservation. 

Computing Robust Strategies for Managing Invasive Plants
Andreas Lydakis, Jenica Allen, Marek Petrik and Tim Szewczyk

Recognition for Camera Traps in Unfamiliar Territory
Sara Beery, Grant Van Horn and Pietro Perona

Crowdsourcing mountain images for water conservation
Darian Frajberg, Piero Fraternali and Rocio Nahime Torres

Exploiting Data and Human Knowledge for Predicting Wildlife Poaching
Swaminathan Gurumurthy, Lantao Yu, Chenyan Zhang, Yongchao Jin, Weiping Li, Xiaodong Zhang and Fei Fang

Towards Automatic Identification of Elephants in the Wild
Matthias Körschens, Björn Barz and Joachim Denzler

Deep Reinforcement Learning for Green Security Game with Online Information
Lantao Yu, Yi Wu, Zheyuan Ryan Shi, Rohit Singh, Lucas Joppa and Fei Fang

Designing the Game to Play: Optimizing Payoff Structure in Security Games
Zheyuan Ryan Shi, Ziye Tang, Long Tran-Thanh, Rohit Singh and Fei Fang

Counting Caribou from Aerial Imagery
Evan Shelhamer, Nathan Pamperin and Trevor Darrell

An agent-based model of an endangered population of the Arctic fox from Mednyi Island
Angelina Brilliantova, Anton Pletenev, Liliya Doronina, and Hadi Hosseini

The Great Grevy’s Rally: A Review on Procedure
Jason Parham, Charles Stewart, Tanya Berger-Wolf, Daniel Rubenstein and Jason Holmberg

Convolutional Neural Networks for Detecting Great Whales from Orbit in Multispectral Satellite Imagery
Patrick Gray and David Johnston

Green Security Game with Community Engagement
Taoan Huang, Rohit Singh and Fei Fang

Simulation for Wildlife Conservation with UAVs
Elizabeth Bondi, Debadeepta Dey, Ashish Kapoor, Jim Piavis, Shital Shah, Fei Fang, Bistra Dilkina, Robert Hannaford, Arvind Iyer, Lucas Joppa and Milind Tambe

Ruling the Roost with CNNs: Detecting and Tracking Communal Bird Roosts in Weather Radar Data
Zezhou Cheng, Saadia Gabriel, Pankaj Bhambhani, Daniel Sheldon, Subhransu Maji, Andrew Laughlin and David W. Winkler

Probabilistic Inference with Generating Functions for Population Models
Kevin Winner, Daniel Sheldon and Debora Sujono

Inferring Latent Velocities from Weather Radar Data using Gaussian Processes
Rico Angell, Daniel Sheldon and Eric Johnson]]> https://docs.microsoft.com/archive/blogs/jennifer/computer-vision-talks-at-icml-2018 Fri, 20 Jul 2018 21:27:33 GMT https://blogs.msdn.microsoft.com/jennifer/?p=3045 The 35th International Conference on Machine Learning (ICML) was held in Stockholm on July 10-15, 2018.  Links to the associated papers and video recordings (when they are posted) will be available on the website under each individual session. 

Overall, deep learning and reinforcement learning were still the hottest topics.  During each session slot, there was a reinforcement learning track and at least one deep learning (neural network architectures) track (and sometimes multiple deep learning tracks).  Here is the breakdown of submitted and accepted papers, which clearly shows the popularity of these topics. 

Talks on the fairness of machine learning algorithms continue to rise.  One of the "best paper" awards went to a paper on delayed impact of fair machine learning.  This was an interesting paper comparing different methods of fairness – demographic parity, equality of opportunity, and unconstrained utility maximization – and introduced the “outcome curve”, a tool for comparing the delayed impact of fairness criteria.  They showed that fairness criteria may cause harms to groups they intended to protect if you consider the long-term effects. 

There was a decent amount of work in generative adversarial attacks. The first keynote on AI & Security and the other “best paper” award were on this topic, as well as some tracks.  The “best paper” winner was “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples”.  This paper examined the non-certified white-box-secure defenses against adversarial examples from ICLR 2018, and found that 7 of the 9 defenses relied on obfuscated gradients.  Their paper developed 3 attack techniques which circumvented 6 defenses completely and 1 partially.  With this, they argued that future work should avoid relying on obfuscated gradients, and also spoke about the importance of reevaluating others’ published results. 

Of particular interest to me this year were the Computer Vision talks.  Here is a quick summary of some of the innovation in Computer Vision.  There were 2 computer vision tracks, on Wed July 11 and on Fri July 13. 

Deep Predictive Coding Network for Object Recognition (link to paper)
They described a bi-directional and recurrent neural net, namely deep predictive coding networks (PCN), that has feedforward, feedback, and recurrent connections. Feedback connections from a higher layer carry the prediction of its lower-layer representation; feedforward connections carry the prediction errors to its higher-layer. Given image input, PCN runs recursive cycles of bottom-up and top-down computation to update its internal representations and reduce the difference between bottom-up input and top-down prediction at every layer. After multiple cycles of recursive updating, the representation is used for image classification. With benchmark datasets (CIFAR-10/100, SVHN, and MNIST), PCN was found to always outperform its feedforward-only counterpart: a model without any mechanism for recurrent dynamics, and its performance tended to improve given more cycles of computation over time. In short, PCN reuses a single architecture to recursively run bottom-up and top-down processes to refine its representation towards more accurate and definitive object recognition. 

Gradually Updated Neural Networks for Large-Scale Image Recognition (link to paper)
Neural networks keep getting deeper, traditionally by cascading convolutional layers or building blocks.  They present a new way to increase the depth: computational orderings to the channels within convolutional layers or blocks.  This not only increases the depth and learning capacity with the same amount of computational cost and memory, but also eliminates the overlap singularities resulting in faster convergence and better performance.  They use “GUNN” for an acronym. 

Neural Inverse Rendering for General Reflectance Photometric Stereo (link to paper)
Photometric stereo is the problem of recovering 3D object surface normals from multiple images observed under varying illuminations.  They propose a physics-based unsupervised learning approach to general BRDF photometric stereo where surface normals and BRDFs are predicted by the network and fed into the rendering equation to synthesize observed images.  This learning process doesn’t require ground truth normals; using physics can bypass the lack of training data. SOTA results outperformed a supervised DNN and other classical unsupervised methods. 

One-Shot Segmentation in Clutter (link to paper)
This was an interesting look at visual search.  They cited “Where’s Waldo?” as a fun example of solving a problem with only one example.  :)  They tackled the problem of one-shot segmentation: finding and segmenting a previously unseen object in a cluttered scene based on a single instruction example.  The MNIST of one-shot learning is the omniglot dataset, and they proposed a novel dataset called “cluttered omniglot” which used all characters but dropped them on top of each other in different colors. Using an architecture combining a Siamese embedding for detection with a U-net for segmentation, they show that increasing levels of clutter make the task progressively harder.  In this kind of visual search task, detection and segmentation are two intertwined problems, the solution to each of which helps solving the other.  They tried a pre-segmenting characters approach.  After segmenting using color, performance got very good.  They introduced MaskNet, an improved model that attends to multiple candidate locations, generates segmentation proposals to mask out background clutter, and selects among the segmented objects (segment first, decide later).  Such image recognition models based on an iterative refinement of object detection and foreground segmentation may provide a way to deal with highly cluttered scenes.  http://github.com/michaelisc/cluttered-omniglot

Active Testing: An Efficient and Robust Framework for Estimating Accuracy (link to paper)
Supervised learning is hungry for annotated data.  There are many approaches for dealing with the lack of labelled data in training (unsupervised, semi-supervised, etc).  Assemble a small high-quality test dataset.  The gold standard: given a fixed budget, annotate "all we can afford".  Their approach: Trade annotation accuracy for more examples.  They reformulate the problem as one of active testing, and examine strategies for efficiently querying a user so as to obtain an accurate performance estimate with minimal vetting.  They demonstrate the effectiveness of their proposed active testing framework on estimating two performance metrics, Precision@K and mean Average Precisions, for two popular Computer Vision tasks, multilabel classification and instance segmentation, respectively.  They further show that their approach is able to significantly save human annotation effort and more robust than alternative evaluation protocols.   

Noise2Noise: Learning Image Restoration without Clean Data (link to paper)
This is interesting for those working with grainy images.  They train a denoiser.  They apply basic statistical reasoning to signal reconstruction by machine learning - learning to map corrupted observations to clean signals - with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption.  In practice, they show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans - all corrupted by different processes - based on noisy data only.

Solving Partial Assignment Problems using Random Clique Complexes (link to paper)
This is an interesting paper to read if you need to perform a matching task, finding the same image (like a building) with occlusion, rotation, etc.  They present an alternate formulation of the partial assignment problem as matching random clique complexes, that are higher-order analogues of random graphs, designed to provide a set of invariants that better detect higher-order structure. The proposed method creates random clique adjacency matrices for each k-skeleton of the random clique complexes and matches them, taking into account each point as the affine combination of its geometric neighborhood.  They justify their solution theoretically, by analyzing the runtime and storage complexity of their algorithm along with the asymptotic behavior of the quadratic assignment problem (QAP) that is associated with the underlying random clique adjacency matrices. 

Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data (link to paper)
Future predictions on sequence data (e.g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics. Context-free grammars are natural choices to capture such properties, but traditional grammar parsers (e.g., Earley parser) only take symbolic sentences as inputs. This paper generalizes the Earley parser to parse sequence data which is neither segmented nor labeled. This generalized Earley parser integrates a grammar parser with a classifier to find the optimal segmentation and labels, and makes top-down future predictions. Experiments show that this method significantly outperforms other approaches for future human activity prediction.

Neural Program Synthesis from Diverse Demonstration Videos (link to paper)
Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans.  For example, learning how to make fried rice from watching a bunch of YouTube videos; humans understand variations like brown or white rice, etc.  To empower machines with this ability, they propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos.  Their model uses 3 steps: extract unique behaviors (using CNNs feeding into an LSTM), summarize (compare demo pairs to infer branching conditions, using multi-layer perceptron, to improve the network’s ability to integrate multiple demonstrations varying in behavior), and decode.  They also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training.  They show that their model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations.  Performance got better with the number of input videos.  The code is available at https://shaohua0116.github.io/demo2program.

Video Prediction with Appearance and Motion Conditions (link to paper)
Video prediction aims to generate realistic future frames by learning dynamic visual patterns. One fundamental challenge is to deal with future uncertainty: How should a model behave when there are multiple correct, equally probable futures? They propose an Appearance-Motion Conditional GAN to address this challenge. They provide appearance and motion information as conditions that specify how the future may look, reducing the level of uncertainty. Their model consists of a generator, two discriminators taking charge of appearance and motion pathways, and a perceptual ranking module that encourages videos of similar conditions to look similar. To train their model, they developed a novel conditioning scheme that consists of different combinations of appearance and motion conditions. They evaluate their model using facial expression and human action datasets – transforming input faces into different emotions with motion/video (generative videos).  They showed one interesting bug: Trump’s eyebrows turned black because no training data had white/blond eyebrows.  You can see this at http://vision.snu.ac.kr/projects/amc-gan, and the code is coming soon to https://github.com/YunseokJANG/amc-gan. 

Rationale

To begin, why would you want to do this?  Here’s the value prop:

• Data Science Virtual Machine: the DSVM enables rapid development for data scientists.  Say that you usually use { TensorFlow | CNTK | PyTorch | etc } for your deep learning framework, but you found a great sample in one of the other deep learning frameworks.  You’d like to quickly try your data with that sample code, but don’t want the overhead of getting a new deep learning framework with all of its dependencies set up on your machine.  Use the DSVM!  It has a huge list of popular machine learning tools already installed and configured.  It is also great for scaling out training of your models on VMs. 
• GPU: I’m not going to do an in-depth processor comparison here, but essentially GPUs have great parallel-processing capabilities and can perform faster than CPUs in many batch-processing/data-intensive scenarios.  For example, I ran the two following commands – the first with GPU support and the second with CPU only – to train a simple machine learning model in PyTorch, and you can see the resulting speedup from about 56 minutes with CPU to less than 16 minutes with GPU. 

GPU: 15m38.556s

th train.lua -input_h5 data/tiny-shakespeare.h5 -input_json data/tiny-shakespeare.json

CPU: 55m51.655s

th train.lua -input_h5 data/tiny-shakespeare.h5 -input_json data/tiny-shakespeare.json -gpu -1

Data Science Virtual Machine versions

There are multiple versions of the Data Science Virtual Machine.  There are similar tools on all of them, but for example, the Ubuntu image contains additional deep learning frameworks that aren’t supported on Windows. 

• Windows Server 2016 DSVM
• Windows Server 2012 DSVM
• Ubuntu DSVM
• CentOS DSVM
• Deep Learning VM (DLVM): this is a variant of the DSVM with GPU support ready to go (so you don’t need to understand the tips & tricks below!).  It is available with a Windows Server 2016 or Ubuntu base image.  The DLVM actually uses the same core VM images as the DSVM, but the main differences are that the setup wizard is optimized for easy provisioning on GPU and the DLVM auto-downloads a set of end-to-end deep learning samples from GitHub when the VM instance is created.
• Geo AI DSVM: this is a Windows Server 2016 DSVM with extra support for geospatial analytics.  It comes preinstalled with ESRI’s ArcGIS Pro software and several geospatial code samples.  

Tips & tricks

To get GPU support, you need both hardware with GPUs in a datacenter, as well as the right software – namely, a virtual machine image that includes GPU drivers so you can use the GPU. 

The biggest tip is to use the Deep Learning Virtual Machine!  The provisioning experience has been optimized to filter to the options that support GPU (the NC series – see below), which make it easier to set it up correctly. 

Outside of the Deep Learning Virtual Machine, the big gotchas to creating a vanilla Data Science Virtual Machine for deep learning on GPU are:

• The virtual machines with GPU support are listed at https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-gpu.  For machine learning, the various versions of NC or ND series hardware is your best bet.  But it is not available everywhere.  Check the list of which datacenters have which hardware on the extremely-useful matrix on the Azure Products by Region Page under Compute, Virtual Machines. 

NOTE: this is a screenshot, so it might not be accurate for you, future reader!  I also only had the US and Canada regions selected, and there are many more datacenters available.  Click here to change the region filters and get the latest data. 

• You need to use an image with GPU drivers installed.  As documented here, GPU drivers are provided on the following machines: Linux (Ubuntu), Linux (CentOS), Windows 2016, and the Deep Learning VM.  The Windows 2012 DSVM does not have GPU support. 
• You need to understand if you need HDD or SSD.  Even though solid state drives seem “better”, not all GPU machines support them.  The different VM series have different requirements for their Azure storage disk support.  NC and NV VMs only support VM disks that are backed by Standard Disk Storage (HDD).  NCv2, ND, and NCv3 VMs only support VM disks that are backed by Premium Disk Storage (SSD).

Connecting to the DSVM

If you are using a Windows data science virtual machine, once the DSVM is provisioned, you can remote desktop into it. 

If you are using a Linux data science virtual machine, once the DSVM is provisioned, you have a couple of choices on how to connect to it.  More details are here, but the quick summary is that you can use any of these options:

• For terminal/console sessions: use SSH.  In the Azure portal, after provisioning your VM, you can click on the “Connect” button to get the exact ssh command.  If you have Bash on Windows support*, you can use ssh right from the Bash app in Windows.  Otherwise, you can download a third-party tool like Putty.   
• For graphical sessions: use the X2Go client. 
• For Jupyter notebooks, use JupyterHub by browsing to https://your-vm-ip:8000 or JupyterLab by browsing to https://your-vm-ip:8000/lab (fill in the appropriate IP address for your virtual machine). 

* In Windows 10, you can enable the Windows subsystem for Linux by running this PowerShell command as administrator and rebooting: “Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux”. 

Resources

Azure Data Science Virtual Machine documentation: don’t forget to explore the whole tree in the left-hand pane 

Data Science Virtual Machine Plans and Pricing: note that this is for the Windows Server 2016 version specifically

Intro to Deep Learning VM

Create a Deep Learning VM

Get the Deep Learning VM

Virtual Machines with GPU support

Get to know your DSVM: shows all of the tools, platforms, utilities, and samples that are included in the Data Science Virtual Machine, neatly organized by category

Data Science Virtual Machine product webpage: this is more of a high-level overview

My keynote is available at https://www.youtube.com/watch?v=CHeXx5IG1T8, and an interview is available at https://www.youtube.com/watch?v=U2Botu4oE6k.  

Here are some of the highlights that you should know about:

• Azure Machine Learning Packages for Vision, Text, and Forecasting in Public Preview – these Python pip-installable extensions for Azure Machine Learning simplify the common tasks of computer vision, text analytics, and forecasting.
• ML.NET – this cross-platform, proven, and open-source machine learning framework enables classification and regression tasks (with more to come) to empower machine learning in .NET. 
• Project Brainwave – this deep-learning platform is an engine to enable real-time AI leveraging FPGAs on Azure. 
• Cognitive Search – this powerful Azure Search with built-in Cognitive Services allows you to make sense of your data; the JFK files demo is a cool example. 
• Microsoft and Qualcomm vision AI developer kit – this partnership allows you to use the Qualcomm Snapdragon Neural Processing Engine with Azure Machine Learning and Azure IoT Edge; register here for early access. 
• Updates to conversational AI and bots – this includes updates to the Azure Bot Service, Bot Builder SDK v4 Public Preview, general availability of QnAMaker, as well as the release of new experimental projects from Cognitive Services labs such as Project Gesture, Project Personality Chat and Project Conversation Learner. 
• Updates to the Cognitive Services include support for object detection in Custom Vision, a unified Speech service, and Video Indexer updates. 
• Windows ML – a hardware-accelerated inference engine (meaning it can evaluate trained machine learning models) that works across Windows 10 devices; the Contoso Shipping app and Office Word prototype both use Windows ML. 

In addition, there were a wealth of great sessions on AI & ML.  Check out the entire event online at https://channel9.msdn.com/Events/Build/2018/, and I’ve pulled out some of the AI/ML sessions below. 

Overview Sessions from Executive Leadership

• Microsoft AI Overview for Developers
• The Microsoft AI Platform: a State of the Union
• AI @ Microsoft, How we do it and how you can too!

Microsoft Research and AI for Good

• AI for Earth: Analyzing Global Data with Azure
• You Can’t Start a Fire Without a Spark: AI for Accessibility
• How Microsoft AI Defeated Ms Pacman

General Machine Learning

• Introducing ML.NET
• What’s New with Azure Machine Learning
• Demopalooza: Get Started Infusing AI Into Your Applications Today
• Demystifying Machine and Deep Learning for Developers

Cognitive Services (using pretrained or customizable models for common artificial intelligence tasks)

• 10 Things Developers Need to Know about Building Intelligent Apps
• Notes from the Field on Customizing Your AI using Cognitive Services
• Computer Vision Made Easy: Consume or Build Your Own State-of-the-Art Models and Use Them Everywhere (this focuses on the Custom Vision service)
• Build Intelligent Applications by Infusing Knowledge from the Web
• Cognitive Search: Announcing the smartest enterprise search engine, now with cognitive skills! 
• Speech Services

Conversational AI and Bots

• Conversational AI: What’s New?
• Conversational AI: Best Practices for Building Bots
• Bot Intelligence, Speech Capabilities, and NLU Best Practices
• What’s New with Cortana and the Skills Kit

Deploying and Scaling Machine Learning

• Hyperscale hardware: ML at scale on top of Azure + FPGA (this is the Project Brainwave work!)
• Deep Learning at Scale in Azure
• DevOps for AI: Deploying Everywhere
• AI + IoT: Building, Deploying, and Managing Your Custom AI on the Edge
• Windows AI Platform and the Intelligent Edge

If you are doing geospatial work, you might also be interested in What are Customers Building with New Bing Maps Capabilities? and/or the Azure Maps announcement, as well as our session covering the land cover mapping APIs.  

So what is “AI for Earth”?  Microsoft has publicly committed $50 million USD over 5 years for artificial intelligence projects that support clean water, agriculture, climate, and biodiversity.  But our true vision extends beyond a grant program: our data science team is working with partners and nonprofits to build a set of APIs to transform these environmental issues.  Imagine machine learning models exposed through services which could differentiate between various species of animal for conservation purposes, predict agricultural yields, estimate the probability of floods, super resolve climate predictions, or classify aerial and satellite imagery into actionable maps of natural resources that could empower land use planners to optimize the use of our planet’s scarce resources. 

Our AI for Earth work aligns to three key pillars:

• Access – our grant program provides cloud resources and the ability to do machine learning at scale
• Education – we are committed to helping our grantees be successful with office hours and training resources
• Innovation – we are developing a showcase of lighthouse projects, publishing research, and collaborating with others to expand and grow initial projects.  At Build, we will talk about the private preview of a new AI for Earth API, and discuss how we are enabling others to produce their own APIs dedicated to environmental sustainability.

I’ve already had the opportunity to start working with some of our grantees, and I’m so impressed with the ways they are using machine learning to address climate change, improve agricultural yields, lessen water scarcity, and protect wildlife and Earth’s biodiversity.  I am excited to share their work with you in the coming months.  In addition, we work very closely with several groups in Microsoft Research, including Project Premonition and FarmBeats (you may have heard me speaking and blogging on FarmBeats in the past). 

Similar to my previous role, I will still be doing a mixture of public speaking in addition to my engineering work.  Here are some upcoming conferences (the hyperlinked ones are open to the public) where I will be speaking about AI for Earth in the next few months:

Conference	Date	Location
O’Reilly AI Conference	4/30-5/2	New York, NY
Data Science Conference	5/3-5/4	Chicago, IL
Build	5/7-5/9	Seattle, WA
AI for Earth Summit	5/16-5/18	Redmond, WA
WHIPS	5/20-5/22	Suncadia, WA
Vivatech	5/24-5/26	Paris, France
MLADS	6/11-6/13	Redmond, WA
O’Reilly AI Conference	9/5-9/7	San Francisco, CA

For more information on the AI for Earth program, please visit http://aka.ms/aiforearth.  If you are doing machine learning work in agriculture, water, climate change, or biodiversity, feel free to apply for a grant as well. ]]> https://docs.microsoft.com/archive/blogs/jennifer/microsoft-research-papers-behind-the-cognitive-services Fri, 09 Feb 2018 15:18:02 GMT https://blogs.msdn.microsoft.com/jennifer/?p=2865 I frequently do public speaking on machine learning research at Microsoft, and sometimes I am asked for more information on the work behind some of the cool services we provide.  In this blog post, I want to provide links to some of the Microsoft Research papers behind the Cognitive Services. 

Computer Vision API:

• https://www.microsoft.com/en-us/research/publication/rich-image-captioning-in-the-wild/

Language Understanding Intelligence Service/LUIS (try it at http://luis.ai):

• https://arxiv.org/ftp/arxiv/papers/1409/1409.4814.pdf
• https://www.microsoft.com/en-us/research/publication/fast-and-easy-language-understanding-for-dialog-systems-with-microsoft-language-understanding-intelligent-service-luis/

You can search for additional Microsoft Research publications at https://www.microsoft.com/en-us/research/search/?content-type=publications&sort_by=most-relevant.