Hello again my fellow media development enthusiasts. It is that time once again. Windows 10 UWP apps are more media capable than ever before. My previous lists of Media centric samples for Windows 8 & 8.1 were popular and hopefully helpful so I wanted to make sure to continue the tradition. In this edition I've also included samples for some of the new D2D technologies that can interoperate with the Media stack. While this list may not be extensive hopefully it is as least close. If you find any other samples that I’ve missed please let me know.

Okay, let’s talk about the BackgroundMediaPlayer architecture a bit. Windows Phone 8 (WP8) was a three process architecture. Windows Phone 8.1 (WP8.1) is a two process architecture. I won’t go into how WP8 works. In WP8.1 you have a UI process and a background player process. The background player process is registered as a singleton instance OOPServer. This means that the background player will only be created if one does not already exist in the current app container.

The background player process contains your background audio code. Just like Windows 8 this process is given a PLM exemption that allows it to continue to run in the background while your UI process can be suspended. Once the exemption is registered, management of the background player process is handed off to the media manger. The media manager coordinates closely with the PLM to ensure there is only one active background player process at a given time. This means only one app can be playing audio in the background at any time.

-James

There are a number of different audio capture and playback APIs on Windows Phone 8.x. Unfortunately If you want to get at the raw PCM data and process it in near real time your options are limited. While it is possible to capture raw PCM audio from managed code it is not recommended and likely will not work for high performance audio apps. Audio capture in managed code is very prone to dropouts. While using a sizeable buffer may allow for a reasonable solution it is still not recommended. For more information about why this is a bad idea please see my blog.

Because of the problems inherent with capturing audio from managed code it is recommended to use a pure unmanaged C++ solution. Specifically the Windows Audio Session API (WASAPI) implemented via COM. WASAPI is a very low level and highly performant audio playback and recording API. As of Windows Phone 8.1 audio latencies tend to be in the 50 ms – 100 ms range round trip. Lower latencies may be possible at the risk of audio dropouts. Latency tends to be very hardware / driver dependent. Implementing a WASAPI solution does require some basic knowledge of COM and extensive knowledge of C / C++.

It is possible to implement the necessary WASAPI functionality in a C++ / COM solution and package it in a standard runtime DLL for consumption by higher level managed languages. This is sometimes called a hybrid solution. There are some implementation details with a hybrid solution to be aware of. You should avoid creating an interface that transports the raw audio data from the C++ library to the unmanaged parent. Streaming audio can create a lot of data. Crossing the managed to unmanaged boundary frequently with large amounts of data can cause performance and memory pressure issues. While these issues can be managed we currently recommend that you encapsulate all of the audio capture, decoding and encoding functionality into the C++ library. You can then implement “light”, i.e. simple interfaces that are not called frequently to manage the functionality of the library from managed code.

There are some distinct limitations inherent in the WASAPI. Most notably WASAPI’s sample rate (record and playback) is locked to the default rate and bitness of the mixing engine (exclusive mode is not supported on the phone platform). There is no way to change the default sample rate programmatically. On most devices this rate will be locked to either 44.1 kHz or 48 kHz and the bitness will be either 16 or 32. This is usually acceptable for most applications but can cause problems with specialty applications that require a certain frequency.

To add to the complexity of this limitation WASAPI does not support sample rate conversion. This means that if your particular application requires audio data at a certain sample rate you will need to provide your own converter. At this time Microsoft does not provide an easy to use sample rate conversion API or component. While sample rate conversation is relatively simple to implement it can be computationally expensive and should be avoided if possible.

Here are some links that should help you get started with the WASAPI on Windows Phone 8.x:

Audio Capture and Render APIs for native code for Windows Phone
http://msdn.microsoft.com/en-us/library/windows/apps/jj715884(v=vs.105).aspx

ActivateAudioInterface
http://msdn.microsoft.com/en-us/library/windows/apps/jj731089(v=vs.105).aspx

IAudioRenderClient interface
http://msdn.microsoft.com/en-us/library/windows/desktop/dd368242(v=vs.85).aspx

IAudioCaptureClient interface
http://msdn.microsoft.com/en-us/library/windows/desktop/dd370858(v=vs.85).aspx

-James

Since the release of Windows 8 Developer Preview I have been pushing our Multimedia group to add MIDI support for Windows Store apps. On April 4th, 2014 I achieved victory. This year at //build we announced the preview release of the Musical Instrument Digital Interface (MIDI) APIs for Windows Store apps. With this announcement you can now create Windows Store apps that can interface with the veritable plethora of MIDI keyboards, synthesizers and mixers that have shipped over the last three decades. That’s right you can now add a touch first XAML interface to your Roland JP-6 via any Windows 8.1 device! This means that musicians, audio engineers, DJs and lighting designers can be empowered to create unique experiences with your software while leveraging the full range of Microsoft Windows 8.1 devices.

The new MIDI APIs are not simple wrappers around the classic Win32 MIDI APIs. These new APIs have been built from the ground up. Our intent here is to create a consistent and extensible platform for the future of MIDI app development. We have focused on simplicity and ease of use. We parse the MIDI data for you and offer it up using the standard WinRT object oriented architecture. We hope that this will allow any WinRT programmer to leverage their existing knowledge to quickly come up to speed on these new APIs. That said we haven’t forgotten about all that custom MIDI parsing code you’ve been using in your apps since the dawn of MIDI. We also make the raw data available to you to party on as you see fit.

There are a number of things to consider before writing a new MIDI enabled app or adding MIDI support to an existing app. First and foremost the new MIDI APIs are still in preview (as of this writing). This is why you see “WindowsPreview” as the namespace prefix rather than “Windows”. Because these APIs are still in preview there is a chance that they may change when the final version is released. That said the current preview version of these APIs comes with a “go live” license. In essence we believe that these new APIs are solid and stable enough to enable you to use them in your apps right now. We will almost certainly tweak the APIs based on your feedback, so you may need to mod your app when the final version is released. That said we will continue to support the preview APIs for existing apps until such time that a vast majority of apps have migrated to the final release. So rest assured the MIDI preview APIs are ready to rock.

There are a few other “gotchas” when developing apps using the new MIDI APIs:

1. Due to some technical strangeness you need to make sure to build your app for the platform that you are on. Using “Any CPU” is not going to work. What this means is that if you have a 64 bit installation of Windows you need to change your build configuration to target x64. Typically you can target x86 on an x64 system and be just fine thanks to “Windows on Windows”. Unfortunately with these new MIDI APIs you need to make sure to target the exact version of the OS. This means x64 on x64, x86 on x86 and of course ARM on ARM. Because of this you are going to need to provide an x86, x64 and ARM version of your app when submitting to the store. Hopefully this will change with the final release.
2. The new MIDI APIs only work on Windows 8.1. So you need to make sure you are using Visual Studio 2013 and target Windows 8.1. This really shouldn’t be much of an issue as the migration from Windows 8 to Windows 8.1 is seamless and easy.
3. You also need to make sure to download the new APIs via NuGet. Visual Studio 2013’s NuGet package manager (under the “project” menu) allows you to easily download the APIs and automatically add a reference to your project. That's right, for the first time we've made new APIs available via NuGet! Cool huh.

I hope you are as excited about the advent MIDI support in WinRT as I am. I can’t wait to see the new and innovative MIDI apps you add to the store.

Resources:

Windows Preview API

MIDI Windows Runtime sample

NuGet: Windows Runtime API for MIDI Preview 1.0.0

-James 

"Music is a higher revelation than all wisdom and philosophy." ~ Ludwig von Beethoven

// Load video file
var uri = new Uri("ms-appx:///Assets/bear.mp4");
var vid = await StorageFile.GetFileFromApplicationUriAsync(uri);
 
// Load audio file
uri = new Uri("ms-appx:///Assets/song.mp3");
var aud = await StorageFile.GetFileFromApplicationUriAsync(uri);
 
// Create our output file
StorageFolder storageFolder = KnownFolders.VideosLibrary;
var file = await storageFolder.CreateFileAsync("out.mp4", CreationCollisionOption.ReplaceExisting);
 
// Create our editor
MediaComposition mc = new MediaComposition();
 
// Add our audio and backgropund audio track
mc.Clips.Add(await MediaClip.CreateFromFileAsync(vid));
mc.BackgroundAudioTracks.Add(await BackgroundAudioTrack.CreateFromFileAsync(aud));
 
// Generate the edited file
await mc.RenderToFileAsync(file);

Note: Audio and video assets have been removed from attached project files.

MediaComposition

http://msdn.microsoft.com/en-us/library/windowsphone/develop/windows.media.editing.mediacomposition.aspx

MediaCompositionBackgroundAudio.zip

I am giving a live chalk talk on October 16th for interested English speaking engineers. Anyone can sign up and join the talk for free!

If you or someone you know would benefit from a discussion about integrating audio and video technologies into your apps please join me. All audio and video questions will be answered. Nothing is taboo :)

This is your rare opportunity to pick the brain of a self-described Microsoft media uber-nerd (me).

Space is limited so sign up using this link TODAY!
https://aka.ms/Jdk7j0

------------------------------------------------

Chalk Talk for Developers: Light up Your Apps with Sights and Sounds

“The sound and music are 50% of the entertainment…” – George Lucas

In this talk we will take you on a tour of the amazing APIs available to enable you to light up your apps with sound and video. From playing a simple video cut scene to adding rich music and sound effects to your next great game, this highly technical talk will cover it all. After the tour we will allow you, the participant, to guide the discussion. What do you want to know about sound and video on the Windows Store app platform that will allow you to take your app to the next level?

Please join us on Wednesday October 16th, 2013 from 11am-1pm PST.  We’ll be hosting a live chalk talk with technical experts from our developer support team.

We’ll look at demos of real-world scenarios, how the technology works, best practices for implementation, and development troubleshooting tips. There will also be plenty of time for you to ask your questions.

If you are interested, sign up today as space is limited.

WHAT: Light up Your Apps with Sights and Sounds

WHEN: October 16th, 2013 11am-1pm PST

WHERE: Online Meeting

REGISTRATION: email Wsdsctreg@microsoft.com with the subject “October 16th Chalk Talk”.

COST: Free

-----------------------------------------------------

Please note: This post is "reprinted" from the "Managing Windows Store Developer Solutions" blog with permission from the author (just so happens to be me). 

Original post found here:

http://blogs.msdn.com/b/wsdevsol/archive/2013/03/08/mixed-media-audio-amp-video-windows-store-vs-windows-phone-8.aspx

As everyone knows, with the release of Windows 8 and Windows Phone 8 we have brought the two platforms closer together than ever before. The keyword here is “closer”. There are still some fairly major implementation and terminology differences between the two platforms particularly when working with digital audio and video. Hopefully this article will help you map and understand the media technologies between the two platforms. In this article I’m assuming that you are familiar with the media technologies on one platform or the other and just need some hints how to migrate your code to the other platform.

Audio / Video Playback Technologies

The MediaElement will likely be your “go to” control on both platforms for playing audio and video. There are a few differences though. You can’t add audio or video effects (plug-ins) to the MediaElement on the phone. The DRM implementation is also quite different between the two platforms. If you are not doing anything “fancy” you should be able to port your MediaElement implementation with no problem.

The MediaEngine (different from the MediaElement discussed above) is new for both platforms in version 8. On both platforms the MediaEngine can only be used from C++ and is not currently supported from managed code. The phone also has slightly reduced functionality. On the phone the MediaEngine only supports frame-server mode. Also keep in mind that on the phone hardware acceleration will always be used.

WASAPI and XAudio2 are also supported on both platforms. Unfortunately at this time these audio only technologies can only be called from C++. They are not currently supported from managed code. While the full XAudio2 functionality is supported on both platforms keep in mind that XAudio2 only supports uncompressed PCM audio on the phone. It is extremely difficult to use XAudio2 with compressed media formats on the phone at this time. The phone also only supports a subset of the WASAPI Core Audio APIs as well.

Background Audio

The big difference playing background audio in a Windows Store app vs. Windows Phone 8 app is the use of the BackgroundAudioPlayer on the Phone and the MediaElement / Audio tag for Windows Store apps. The BackgroundAudioPlayer is a static class that can be used without needing instantiation. The MediaElement and Audio tags must be defined in markup before they can be used.

For your Windows Store app you must set MsAudioCategory=”BackgroundCapableMedia”. For your phone app you need to create an AudioPlayerAgent that will run in the background when your app gets swapped out.  

The final thing to note is that on both platforms you need to handle user input from the lock screen media controls. On the phone we call these controls the “Universal Volume Control” (UVC) and in your Store app we call them the “media controls”. Different terms but the same basic functionality.

Sharing Media

The big difference here is that with Windows Store apps the user initiates the sharing process using the “share charm”. On the phone your app will need to invoke the ShareMediaTask directly from within your code.

In both cases you will likely never get confirmation that your data was successfully shared. In essence it’s a “fire and forget” operation.

Keep in mind that on the phone your app will be swapped out and may get shut down when the app you are sharing with takes up the full screen. Because of this you may need to save state.

Follow us on Twitter @wsdevsol.

The reason for this limitation is due to the way that the Media Engine (the underlying component that drives the MediaElement) is being used by the XAML framework. There are a number of modes that the Media Engine can use to deliver frames to a rich compositor such as the XAML framework. The Media Element can work in either frame-server or rendering mode. Frame-server mode allows the Media Engine to offer up individual frames that can be used in a composition chain. Rendering mode allows the Media Engine to render directly on a "DirectComposition" (DComp) surface provided by the application. In the current XAML framework implementation we use DComp for the rendering.

DComp has the benefit of being hardware accelerated and is integrated directly into the DWM. This gives the MediaElement the best possible video playback performance and allows it to have a frame-rate independent of its XAML host.