Using media extensions (HTML)

Article
10/13/2015

[This article is for Windows 8.x and Windows Phone 8.x developers writing Windows Runtime apps. If you’re developing for Windows 10, see the latest documentation]

This topic describes what media extensions are and how to use them in your Windows Runtime app.

Introduction

Media extensions add functionality to the media pipeline in your Windows Runtime app. Some possible extensions are audio and video effects, capture effects, transcoding, Digital Rights Management (DRM), support for new decoders and encoders, or a custom capture sink, to name a few. Some extensions are included in the system, such as the VideoStabilization effect and the DRM MediaProtectionManager. Functionality not included in the system can be added using custom media extensions.

For an in-depth sample that uses custom media extensions, see the Media Extension Sample. This sample shows how to create and enable a custom video gray-scale effect, a custom decoder, and how to use the built in VideoStabilization effect.

Another sample that uses media extensions is the Real-Time communication sample.

Media Pipeline and adding and removing effects and extensions

The Media Foundation Pipeline consists of Media Sources, Media Foundation Transforms, such as video effects, encoders and decoders, and Media Sinks. There are a number of ways to add effects and extensions to a Windows Store app. The MediaExtensionManager allows you to replace the media source and encoders and decoders. Methods on the playback, capture, and transcode objects lets you add and remove effects. The MediaCapture class allows you to add a custom sink.

How to add and remove effects to audio and video playback objects depends on the programming language you are using.

For Windows Runtime app using JavaScript, use msInsertVideoEffect, msInsertAudioEffect, and msClearEffects. These methods are exposed on the HTML5 audio and video tags.

For Windows Runtime apps using C# or Visual Basic, use MediaElement.AddAudioEffect, MediaElement.AddVideoEffect and RemoveAllEffects.

To add and remove effects for transcoding, use the MediaTranscoder.AddVideoEffect and MediaTranscoder.AddAudioEffect, MediaTranscoder.ClearEffects.

To add and remove effects for media capture, use the MediaCapture.AddEffectAsync and MediaCapture.ClearEffectsAsync.

To register scheme handlers, byte-stream handlers, audio encoders, audio decoders, video encoders, and video decoders, use the MediaExtensionManager.

To add a custom sink to media capture, use the MediaCapture.StartRecordToCustomSinkAsync and MediaCapture.StartPreviewToCustomSinkAsync.

DRM is supported through the MediaProtectionManager.

Built-In extension

The Windows Runtime provides a built in effect for stabilizing shaky video, such as video captured from a hand held camera.

The How to add video stabilization describes how to use the VideoStabilization effect in a Windows Runtime app using JavaScript. The Media extension sample also shows how to use the VideoStabilization effect.

Custom extensions

You can use custom media extension to add new functionality to the media pipeline. For example, you could create a custom video effect to convert the video to grayscale, as in the Media extension sample. Or create a custom extension to add support for video and audio formats that are not natively supported by the system.

Creating a custom extension consists of two parts. The first is to write the Media Foundation component and the second is to wrap this component in a Windows Runtime media extension object. For an in-depth sample that creates a number of different custom media extensions, see the Media extension sample.

A Media Extension consists of a hybrid object that implements both Component Object Model (COM) and Windows Runtime interfaces. The COM interfaces interact with the Microsoft Media Foundation pipeline. The Windows Runtime interfaces activate the component and interact with the Windows Store app.

In most situations, it is recommended that you use Visual C++ with Component Extensions (C++/CX ) to interact with the Windows Runtime. But in the case of hybrid components that implement both COM and Windows Runtime interfaces, such as Media Extensions, this is not possible. C++/CX can only create Windows Runtime objects. So, for hybrid objects it is recommended that you use Windows Runtime C++ Template Library to interact with the Windows Runtime. Be aware that Windows Runtime C++ Template Library has limited support for implementing COM interfaces.

Be aware that custom extensions are only accessible to the application they are created and registered in. They are not accessible to other applications.

Hh700365.wedge(en-us,WIN.10).gif High level steps to create a custom effect.