1. If you see an error about missing dll files, then you can download them from https://qemu.weilnetz.de/w32/dll/.
2. You will see a warning saying that the -redir option has been replaced by the -netdev option. I am still using -redir, since I could not understand how to use -netdev. If you know how, please leave a comment below and I will replace the command.

1. Using the above instructions we can emulate an ARM1176 CPU (which is the same as the one on Raspberry Pi 1) that runs a generic kernel on a Raspbian OS. Technically, this is slightly different than emulating a Raspberry Pi 1, since there are differences in the kernel and the board configuration (e.g. GPIO pins). QEMU does provide support for better Raspberry Pi 1 and Raspberry Pi 2 emulation, but it does not support any network capabilities and I could not make it work (the screen remained black and the system did not boot). Andrew Baumann initially implemented the port for Raspberry Pi 2 at https://github.com/0xabu/ and he also helped me with this tutorial. If you want to try by yourselves (and please do so and let me know, if you make it work), then here are the changes that you need to start with, compared to the above instructions:
   • In step #3 you need to use a precompiled kernel for Raspberry Pi 1 or Raspberry Pi 2.
     • One option is to download them https://github.com/raspberrypi/firmware/tree/master/boot
       • kernel.img is the kernel for Raspberry Pi 1
       • kernel7.img is the kernel for Raspberry Pi 2
     • A more difficult option is to install Raspbian on a real SD card, mount it (I'm not sure even if there are tools in Windows to do this) and then copy the kernel from the /boot directory.
   • In order to boot the emulator you need to use the following commands:
     • Raspberry Pi 1: qemu-system-arm -M raspi -kernel kernel.img -sd 2016-09-23-raspbian-jessie.img -append "rw earlyprintk loglevel=8   console=ttyAMA0 root=/dev/mmcblk0p2" -serial stdio
     • Raspberry Pi 2: qemu-system-arm -M raspi2 -kernel kernel7.img -sd 2016-09-23-raspbian-jessie.img -append "rw earlyprintk loglevel=8   console=ttyAMA0 root=/dev/mmcblk0p2" -serial stdio
2. In order to enable SSH, we used the option -redir tcp:2222::22 in the qemu-system-arm command. This option is deprecated and replaced by -netdev, that's why it triggers a warning message in the output. However, I could not understand how the -netdev option works. If you know how, please let me know in your comments, so that I can replace the above command.
3. It is not possible to copy data from the QEMU window, because it is an emulated framebuffer. If you know how to paste data to it, please write it in the comments and I will update the post accordingly.
4. If you have any tips regarding how to improve performance, then please post them in the comments. I tried allocating more than 256MB memory (using the -m 512 option in qemu-system-arm) as well as > 1 CPU cores (using the -smp 4 option), but neither works.
5. If you know of any other emulators/simulators for popular IoT devices (running on desktop, browser, etc), feel free to paste them in the comments section.

1. Overview of the Windows 8.1 audio stack
2. Audio categories
3. H/W offload
4. Audio processing (audio effects)
5. Background audio playback, volume policies and routing

In this post I will explain the differences between the Windows 8.1 audio stack and the Windows 10 audio stack. During the development of Windows 10, I spent a considerable amount of time working with the MSDN doc writers and the audio developers, in order to update the MSDN documentation for audio. So, in order to avoid duplicating information, I will try in this post to provide a quick summary of each new feature and then point you to the appropriate MSDN article, which will go into more depth.

Also, if you like listening to videos more than reading blog posts and documentation, then you're in luck. I worked with Andy Wen from our PC Ecosystem team and we presented an overview of the changes in WinHEC China 2015. You can find our presentation "Enabling Great Audio Experiences in Windows 10" (the video is translated in real-time from Mandarin to English. You can also download the English version of the slides from that page as well).

These are the areas, where the audio stack changed in Windows 10:

1. Low latency Audio
2. AudioGraph: a new API for Media Creation scenarios
3. New signal processing modes and audio categories
4. Offload APOs: APOs can be loaded on top of the offload pin
5. Cortana Voice Activation
6. New mechanism for Background Audio Playback
7. New audio codecs: ALAC, FLAC, AMR-NB
8. Windows Universal Drivers for Audio: The same driver works in both Windows 10 Desktop SKU and Windows 10 Mobile SKU
9. Resource management for audio drivers
10. PNP Rebalance support for audio drivers

Looking at the above list of all changes in the Windows 10 audio stack, I just realized that I owned 80% of those features (I'm including a couple that I co-owned)! Not bad :)

In the following sections I will provide a quick overview of the change and point you to the corresponding MSDN links for additional information.

1. Low latency audio

Audio latency is the delay between that time that sound is created and when it is heard. Having low audio latency is very important for several key scenarios, such as the following.

• Pro Audio
• Music Creation
• Communications
• Virtual Reality
• Games

The total latency of a device is the sum of the latencies of the following components:

• OS
• APO
• Audio driver
• H/W

In Windows 10 we minimized the latency of the OS to a bare minimum. Without any driver changes, all applications in Windows 10 will have 4.5-16ms lower latency. In addition, if the driver has been updated to take advantage of the new low latency DDIs that use small buffers to process audio data, then the latency can be reduced even more. In fact, as is shown in the following diagram, if a driver supports 3ms audio buffers, then the roundtrip latency is ~10ms. And today there are systems in the market that support <3ms buffers :)

 In order to have a deeper dive of the low latency optimizations in the Windows stack, as well as the new low latency mechanisms that we provide for applications and drivers, please look at the Windows 10 low latency whitepaper. Also, you can look at our WASAPI low latency sample in GitHub.

2. AudioGraph

AudioGraph is a very simple API that reduces lots of the overhead in Media Creation scenarios. It simplifies implementation by using the concept of an "audio graph".

An audio graph is a set of interconnected audio nodes through which audio data flows. Audio input nodes supply audio data to the graph from audio input devices, audio files, or from custom code. Audio output nodes are the destination for audio processed by the graph. Audio can be routed out of the graph to audio output devices, audio files, or custom code. The last type of node is a submix node which takes audio from one or more nodes and combines them into a single output that can be routed to other nodes in the graph. After all of the nodes have been created and the connections between them set up, you simply start the audio graph and the audio data flows from the input nodes, through any submix nodes, to the output nodes. This model makes scenarios like recording from a device's microphone to an audio file, playing audio from a file to a device's speaker, or mixing audio from multiple sources quick and easy to implement.

A good starting point to learn more is the MSDN page for AudioGraph. If you want to start experimenting with the AudioGraph API is to use our AudioGraph sample in GitHub.

3. New audio categories and signal processing modes

In my previous posts, I explained the concepts of audio categories and signal processing modes.

The following matrix shows the full list of audio categories in Windows 10:

The following matrix shows the full list of audio processing modes in Windows 10:

Let me clarify a few things between audio categories and audio signal processing modes:

• Applications tag each of their streams with one of the 10 audio categories that are provided by Windows
• Each audio category is mapped to a specific signal processing mode.
• Applications do not have the option to change the mapping between an audio category and a signal processing mode.
• Applications have no awareness of the concept of an "audio processing mode". They cannot find out what mode is used for each of their streams
• Each signal processing mode needs to include audio processing (i.e. APOs) that optimize the audio signal for the scenarios described above.
• However, it is possible that not all modes might be available for a particular system
• Windows only mandate that the driver supports raw and/or default signal processing modes
• Drivers define which signal processing modes they support (i.e. what types of APOs are installed as part of the driver) and inform Windows accordingly
• If a particular mode is not supported by the driver, then Windows will use the next best matching mode
• It is recommended that IHVs/OEMs utilize the new modes to add audio effects that optimize the audio signal to provide the best user experience

The following diagram shows a system that supports multiple modes:

For additional information about audio categories and audio signal processing modes, please visit the MSDN page for "Audio Processing Object Architecture" and "Audio Signal Processing Modes".

4. Offload APOs

In my post about the Windows 8.1 H/W offload, I had written that Windows 8.1 that does not support loading of APOs on top of the offload pin. However, in Windows 10, we added support for this. This change enables some new scenarios, e.g.:

1. In Windows 8.1, if the audio effects cannot be implemented in H/W (e.g. because the DSP is not powerful enough), then there was no practical way for the system to support offload. In Windows 10, these systems would implement the audio effects in S/W, in the form of APOs on top of the offload pin.
2. In Windows 10, we can split the audio processing between S/W and H/W. Also, the processing can change dynamically (i.e. some/all processing might be moved from the S/W APO to the DSP when there are enough H/W resources and then moved back to the S/W APO when the load in the DSP increases)

Here is a diagram showing both the offload pin and the host pin:

For more information about offload APOs, please visit the MSDN page titled "Implementing Hardware Offloaded APO Effects".

5. Cortana Voice Activation

Cortana, the personal assistant technology introduced on Windows Phone 8.1, is now supported on Windows 10 devices. Voice activation is a feature that enables users to invoke a speech recognition engine from various device power states by saying a  specific  phrase  - "Hey Cortana".  The  "Hey Cortana" Voice Activation (VA) feature allows users to quickly engage an experience (e.g., Cortana) outside of his or her active context (i.e., what is currently on screen) by using his or her voice.

You can find more information about this feature at the MSDN page for Voice Activation.

6. Background Audio Playback

 In my previous post about Windows 8.1 background audio playback,I had explained that there were different mechanisms for apps to render audio from the background between the Desktop SKU and the Mobile SKU. In Windows 10, we have 1 mechanism that works for both SKUs and it is similar to the Windows 8.1 Mobile SKU mechanism.

More specifically, a Windows 10 application that wants to play audio from the background needs to consist of two processes. The first process is the main app, which contains the app UI and client logic, running in the foreground. The second process is the background playback task, which implements  IBackgroundTask like all UWP app background tasks. The background task  contains the audio playback logic and background services. The background task communicates with the system through the System Media Transport Controls.

The following diagram is an overview of how the system is designed:

For more information, please visit the MSDN page for Background Audio.

7. Audio codecs

In Windows 10 we added native support for FLAC, ALAC and AMR-NB codecs.

The list of supported audio codecs is:

For more information, please visit the MSDN page for "Supported Codecs".

8. Windows Universal Drivers for Audio

Windows 10 supports one driver model that works for PC and 2:1’s and Windows 10 for phones and small screen tablets. This means that IHVs can develop their driver in one platform and that driver works in all devices (desktops, laptops, tablets, phones). The result is reduced development time and cost.

In order to develop Universal Audio Drivers, you should take advantage of the following tools:

1. Visual Studio 2015: New driver setting to set “Target Platform” equal to “Universal”
2. APIValidator: WDK tool that checks whether driver is Universal
3. Audio samples in GitHub: Samples sysvad and SwapAPO have been converted to Universal Drivers

For more information, please visit the MSDN page for "Universal Windows Drivers for Audio".

9. Resource Management for Audio Drivers

One challenge with creating a good audio experience on a low cost mobile device, is that some devices have various concurrency constraints. For example, it is possible that the device can only play up to 6 audio streams concurrently and supports only 2 offload streams. When there is an active phone call on a mobile device, it is possible that the device supports only 2 audio streams. When the device is capturing audio, the device can only play up to 4 audio streams.

Windows 10 includes a mechanism to express concurrency constraints to insure that high-priority audio streams and cellular phone calls  will be able to play.   If the system does not have enough resources, then low priority streams are terminated.  This mechanism is only available in phones and tablets not on desktops or laptops.

For more information, please visit the MSDN page for "Audio Hardware Resource Management"

 10. PNP Rebalance for Audio Drivers 

For more information, please visit the MSDN page for "PNP rebalance for Audio drivers".

MSDN References:

1. Windows 10 low latency whitepaper
1. WASAPI low latency sample in GitHub
2. AudioGraph
1. AudioGraph sample in GitHub
3. Audio Processing Object Architecture
1. Audio Signal Processing Modes
4. Implementing Hardware Offloaded APO Effects
5. Voice Activation
6. Background Audio
7. Supported Codecs
8. Universal Windows Drivers for Audio
1. Audio samples in GitHub
9. Audio Hardware Resource Management
10. PNP rebalance for Audio drivers

1. Background Audio Playback
2. Volume/Ducking
3. Routing

1. Background Audio Playback

This policy defines, if the application will keep playing audio when the application goes to the background (e.g. because it is minimized). The application's behavior will differ based on whether it is a Win32 or a Store app. Also, Store apps have different behavior, if they are running on the desktop or on the phone. The following matrix explains the differences:

2. Volume/Ducking Policy

At any given point in time, there might be lots of streams running in a system. Each stream could have a different audio category. For example, let's assume that a user receives a Skype call, as he is listening to music.

1. Xbox Music stream is using the BackgroundCapableMedia category
2. Skype stream is using the Communications category

One common misconception about the Windows audio stack is that Windows natively includes audio effects that modify the audio signal, e.g. acoustic echo cancellation, noise reduction, gain control, etc. However, in reality the audio stack just passes the unmodified raw signal from the application to the speakers or from the microphone to the application. The signal is not modified by any Windows component. In fact, the audio stack loads external components (called "Audio Processing Objects" or APOs) that modify the audio signal. There are multiple companies that create APOs for Windows, such as Dolby, DTS, Waves, Conexant, ForteMedia, Realtek, etc. All commercial systems for the well-known OEMs include at least one APO in the system. It is up to the OEM to decide what type of processing they want in each model and which company will create the APO for that model. In Windows 8.1 the APOs are bundled and installed together with the audio driver.

All APOs are required to declare the type of processing that they do, so that the applications can be informed about the type of processing that is available in each system. Windows provides a list of 18 types of audio effects (as is shown in this MSDN link):

1. Acoustic Echo Cancellation
2. Noise Suppression
3. Automatic Gain Control
4. Beam Forming
5. Constant Tone Removal
6. Equalizer
7. Loudness Equalizer
8. Bass Boost
9. Virtual Surround
10. Virtual Headphones
11. Speaker Fill
12. Room Correction
13. Bass Management
14. Environmental Effects
15. Speaker Protection
16. Speaker Compensantion
17. Dynamic Range Compression
18. Other

1. For playback processing: Control Panel -> Sound -> Playback tab -> Right click on speakers -> Properties -> Enhancements
2. For capture processing: Control Panel -> Sound -> Recording tab -> Right click on microphone -> Properties -> Enhancements

Alternatively, if you don't want to go to control panel, then you can right lick on the speaker icon (at the bottom right of the screen) and select "Playback devices" (opens the "Playback tab" directly) or "Recording devices' (opens the "Recording tab" directly).

 

1. An individual stream: Stream effects (SFX)
2. All streams that use same mode (explained below): Mode effects (MFX)
3. All streams that use the same endpoint (e.g. speakers, microphone, etc): Endpoint effects (EFX)

Each of the 18 audio effect types from the above list can be applied to any of the 3 positions (SFX, MFX, EFX). Also, each position (SFX, MFX, EFX) can have processing for multiple effect types.

Audio effects can be implemented in S/W, in H/W or a combination of both.

If you want more information about this topic, you can look at the Audio Processing Object Architecture page in in MSDN. Just note that the MSDN page is describing the Windows 10 architecture, which is slightly different than the Windows 8.1 architecture that is described in this post.

Overview

As I mentioned in my previous blog post, one of the policies that is determined based on the audio categories is related to power savings. In order to lower power consumption for audio, we need to wake up the CPU less frequently and put less load on the CPU (by offloading more functionality to H/W that is dedicated to audio, such as the audio codec or the audio DSP). This feature is called Hardware-Offloaded Audio Processing (HAP) or simply audio offload.

Audio buffer sizes in Windows 8.1

In order to explain this functionality better, it makes sense to start by explaining the first step while streaming audio: the application writes the audio data in a buffer, so that it can be processed by the audio stack. There are 2 available buffer sizes:

1. 10 ms (this is called the “host pin” or the “system pin”)

   1. The CPU wakes up every 10ms, fills in the 10ms buffer and can go to lower power states.

2. 1 second (this is called the “offload pin”)

   1. The CPU wakes up every 1sec, fills in the 1-second buffer and can go to lower power states.

There is a trade-off between using each of the two buffer sizes.

The 10ms buffer leads to:

1. Lower latency (imagine that if the application fills in the buffer and immediately needs to play “ding” tone, then it only needs to wait for 10ms until the next pass),

2. High power consumption (we need to wake up the CPU every 10ms to write data into the audio buffer)

In general, we want to use this buffer for anything that requires low latency, such as VOIP calls, alerts, sound effects, etc.

The 1 second buffer leads to:

1. High latency (after the data is written into the buffer, we need to wait 1 second before being able to write the next “ding” into the buffer)

2. Low power consumption (we wake up the CPU every 1 second)

In general, we want to use the 1 second buffer, when we play audio files or movies. In that case, we want the battery to last as long as possible and waiting for a few seconds until the beginning of an mp3 file or a movie is not important.

H/W and S/W requirements to support audio offload

Hopefully, with the above use cases, it now makes sense why we’ve tied the 1-second buffers with the usage of the following 2 categories:

1. BackgroundCapableMedia: Mostly used for audio playback

2. ForegroundOnlyMedia: Mostly used for video playback

More specifically, one of the requirements in order to use the 1-second buffers is to use the two categories above. However, things become a little more complicated, when we take into consideration that not all H/W and drivers support this feature, and when we include both “screen-on” and “screen-off” scenarios.

So, here are the requirements, in order for an application to use the 1-second buffer, when the screen is ON:

1. The application needs to be a Store application (offload is not available to Win32 apps. I will explain the reasons below)

2. The application needs to set the audio category of a stream as BackgroundCapableMedia or ForegroundOnlyMedia

3. The H/W needs to be offload-capable (either the DSP or the audio codec)

4. The driver needs to support audio offload.

   1. Note: Even though I have not talked much about audio drivers yet, I’d like to point out that only that use the WaveRT audio miniport model can support audio offload

In addition to the all the above requirements, in order for an application to use the 1-second buffer, when the screen is off  (also known as Low Power Audio or LPA), the following requirements need to be met:

1. The application needs to set the audio category of a stream as BackgroundCapableMedia

2. The application needs to declare in its manifest that it is background-capable (for more information: https://msdn.microsoft.com/en-us/library/windows/apps/hh700367.aspx)

3. The H/W needs to support the Connected Standby (CS) low power state

Intel has a whitepaper for LPA at https://software.intel.com/en-us/articles/low-power-audio-playback-windows-store-whitepaper

Offload diagram

Apart from waking up the CPU less often (which is achieved by using a larger buffer), in order to achieve lower power consumption, we also need to ask the CPU to do less work. In the diagram that I included in my previous post, I showed that the audio data is passed from the application to the Audio Device Graph (audiodg). AudioDG loads the Audio Engine (AudioEng), which allows 3^rd party dlls called APOs (Audio Processing Objects) to process the audio stream. Actually, this is the correct path when we use 10ms buffers. However, for 1-second buffers, Windows expects that all the processing will be done in H/W. As a result, for Windows 8.1, we do not load APOs, when we use the offload pin (1-second buffers). Here is a diagram that shows this:

Here I wanted to clarify that audio effects can be implemented:

1. In S/W: Audio Device Graph (audiodg) loads 3^rd party dlls, which are called Audio Processing Objects

   1. This option is only available, when there is no offload (i.e. for 10ms buffers)

2. In H/W: By the audio codec or the DSP

   1. This option is available regardless of whether we use offload (1 second buffers) or not (10ms buffers)

FAQ: Why is offload not supported for Win32 apps?

One more point that I’d like to make here is that Windows only requires drivers to support 2 offload pins. It’s up to H/W developers to determine, if they want/can support more. Each offload pin corresponds to one stream that can be offloaded. So, if the H/W has 2 offload pins, this means that 2 streams can be offloaded (i.e. use the 1-second buffer) at any point in time. The 3^rd stream (and all streams after that) that tries to use the offload pin will actually be switched to the non-offloaded path (10ms buffers). Implementing an offload pin takes a lot of H/W resources, that’s why most H/W cannot simultaneously support many (i.e. 3+) offloaded streams.

So, this makes it easier to understand why we offer support for offload only to Store apps and not to Win32 apps. Most Store apps are not background-capable, so we can suspend their audio streams when they go into the background (or are minimized, etc). This means that the offload pin that they occupied is now available for other apps to use. However, Win32 apps do not get suspended when they go into the background. So, if a Win32 application, such as Windows Media Player is sitting in the background with a paused stream, we cannot take its offload pin and make it available to other apps. Since most Win32 apps have a long process lifetime (i.e. users let them run in the background), if we allowed them to offload audio streams, then they would easily grab the two offload pins and keep them unused for long periods of time.

FAQ: Can a system have only S/W audio effects and support audio offload?

The last topic that is often asked from H/W manufacturers is whether the following combination is possible:

1. Support audio offload

2. Implement audio effects in S/W (as APOs that are loaded by audiodg)

3. Do not implement any audio effects in H/W

Unfortunately, this combination is not possible in Windows 8.1. All systems that support audio offload, need to have their audio effects implemented in H/W. If the audio effects were implemented as APOs that are loaded by audiodg, then sometimes they would be applied on the audio stream (when the audio stream is going through audiodg) and sometimes they would not be applied (when the audio stream is being offloaded). As a result, the user would see inconsistent behavior (the same file would sound differently based on whether the stream is offloaded or not).

If a H/W manufacturer cannot implement the audio effects in H/W and still wants to support audio offload, then he can choose to implement the audio effects in kernel-mode (loaded by the audio drivers). This is a more complicated process, as it complicates driver development, but this option is provided by Windows.

Additional resources:

1. MSDN link for Audio offload: https://msdn.microsoft.com/en-us/library/windows/hardware/dn302038%28v=vs.85%29.aspx

2. MSDN link for Low Power Audio (LPA): https://msdn.microsoft.com/en-us/library/windows/hardware/dn621143(v=vs.85).aspx

3. Intel’s whitepaper for Low Power Audio: https://software.intel.com/en-us/articles/low-power-audio-playback-windows-store-whitepaper

• Games might have soft music playing in the background, as well loud game effects (e.g. when a user fires a bullet)

• Media players play audio and video files

• Communication applications (e.g. Skype, Lync) allow users to talk to each other remotely

• Sound recorders record audio from other applications or from the environment

   

In order to inform the system about the usage of an audio stream, applications have the option to tag the stream with a specific “audio category”.  Applications can set the audio category, using any of the audio APIs, just after creating the audio stream.

Windows 8.1 supports the following 8 audio categories:

1. ForegroundOnlyMedia: Games or other sounds designed to work only in the foreground, but will mute existing background media sounds. Examples include game audio needed for a game (dancing games, music games), feature films (designed to pause when they go to the background)

2. BackgroundCapableMedia: For audio that needs to continue playing in the background. Examples include: Local media playback, local playlist, streaming radio, streaming playlist, music videos, streaming audio/radio (e.g. YouTube, Netflix, etc)

3. Communications: For audio streaming communication audio such as Voice over IP (VoIP), real-time chat or other type of phone call

4. Alerts: Looping or longer running alert sounds, such as alarm, ring tones, ringing notification, sounds that need to decrease existing audio

5. SoundEffects: Sounds designed to mix with existing audio, such as beeps, dings, and other brief sounds

6. GameEffects: Game sound effects designed to mix with existing audio, such as balls bouncing, engine sounds, characters talking, all non-music sounds

7. GameMedia: Background music played by a game

8. Other: Default stream category used for uncategorized streams

This information allows the audio stack to optimize the user experience. More specifically, the stream audio category defines several characteristics of the way that the system interacts with the audio stream, such as:

1. Audio routing

   • Which endpoint should we use to play or capture sounds related to that stream?

   • Example: When a user connects/disconnects new device (e.g. headset, Miracast, BT, USB, etc), should a particular audio stream switch to it or not?

2. Volume

   • When a user starts a new stream, do we need to attenuate/mute any of the streams that are already playing or capturing audio?

   • Example: When a user starts speaking on Skype, it is ok if a YouTube video keeps playing in the background at full volume?

3. Audio processing

   • What types of audio effects should the system apply to each audio stream?

   • Example: communication streams might need not only to reduce unneeded ambient noises by enabling Acoustic Echo Cancelation and Noise Suppression, but also to capture the user’s voice clearly by enabling Beam Forming and Acoustic Gain Control.

4. Power savings (via H/W offload)

   • Should the processing of the stream be offloaded to the H/W, so that we can lower power consumption of the device?

   • Example: When a user watches a movie, the system has the option to prefetch big audio buffers and send them to the H/W for processing, in order to avoid waking up the CPU very often

5. Background Audio Playback

   • When the application goes into the background, should audio keep playing?

   • Example: When a user is playing a game and he minimizes the window, in order to open a new tab in Internet Explorer, should the audio from the game still be audible?

In the next few posts I will dive deeper into each of the 5 policies mentioned above.

Code samples: How to set an audio category

1. XAML: MediaElement

<MediaElement x:Name="mediaplayer"  AudioCategory="Communications" />

2. HTML: Audio Tag

<audio msAudioCategory="Communications“>

3. C++: WASAPI

// Instantiate the interface to set the audio categories

IAudioClient2 *pAudioClient2;

pMMDevice->Activate(__uuidof(IAudioClient2), CLSCTX_ALL, NULL, (void **)&pAudioClient2));

// Set the category of the stream to be Communications

AudioClientProperties props = {};

props.cbSize = sizeof(props);

props.eCategory = AudioCategory_Communications;

pAudioClient2->SetClientProperties(&props);

From a high level perspective, the audio stack has 6 main components:

1. APIs

1. High-level APIs:

   1. Supported APIs:

      1. XAML MediaElement (C#, VB, C++)

      2. HTML <audio> and <video> tags (used by websites and Windows Web Apps)

      3. Media Foundation (C++)

      4. Windows.Media.Capture (C#, VB, C++)

   2. Deprecated APIs:

      1. DirectShow

      2. DirectSound

      3. PlaySound

      4. Windows.Media.MediaControl

2. Low-level APIs:

   1. Recommended:

      1. For Streaming:

         1. WASAPI (high performance, but more complicated)

         2. XAudio2 (games)

         3. MIDI

      2. For Device Enumeration:

         1. Windows.Devices.Enumeration

   2. Not recommended for Windows applications:

      1. MMDevice API (replaced by Windows.Devices.Enumeration)

      2. DeviceTopology API

      3. EndpointVolume API

2. Audio Device Graph (audiodg.exe), which loads the Audio Engine (audioeng.dll)

   1. Corresponds to Android’s AudioFlinger

   2. Mixes and processes audio streams

   3. Loads “Audio Processing Objects” (APOs), which are H/W-specific plugins that process the audio signal. Android has a similar element called “audio effects”

3. Audio Service (audiosrv.dll)

   1. Used to setup and control audio streams

   2. Implements Windows policies for background audio playback, ducking, etc

4. Audio Endpoint Builder (audioendpointbuilder.exe)

   1. Used to discover new audio devices and create S/W audio endpoints

5. Audio drivers

   1. They follow the port-miniport model (corresponds to the Advanced Linux Sound Architecture - ALSA)

   2. Allow the audio stack to render and capture audio from several audio devices, including: integrated speakers and microphones, headsets/headphones, USB devices, Bluetooth devices, HDMI, etc

6. H/W

   1. Audio codec

   2. DSP (optionally)

   3. Integrated speakers, microphone, etc

   4. External devices: USB audio devices, Bluetooth audio devices, HDMI audio, etc

   5. Signal processing can also be implemented in the H/W (e.g. the codec or the DSP), instead of or in addition to the APOs

The following diagram shows a graphical view of all the above items:

In the following blog posts I will dive deeper into each of the audio components that were described above.

Finally, I would like to thank Frank Yerrace and Kishore Kotteri from the Audio dev team for their contributions to this article.

After 3 years of absence, I returned to Microsoft! I'm back as a PM in the Windows team, working on audio. My team owns the whole audio stack, from the high-level audio-related APIs to the audio driver stack. It feels great to be back in such a great company and an amazing team! Anytime that you hear a sound from your Windows device, you're utilizing my team's hard efforts :) My teammate Matthew van Eerde also has a blog with great information about Windows audio.

From this point on, I'll be mostly focusing on posts related to Windows audio (even though I'll also try to include posts about driver development). As always, I'm open for questions and feedback! Especially if you have feedback about areas that are broken and need our attention, this is the best time to let us know!

I'm really excited to be back! My goal is to keep up with useful blogs posts, as before.

So Long, Farewell, Auf Wiedersehen, Good-bye...

Welcome to the blogosphere USB Team :)