Optimize media resources (XAML)

Article
08/31/2015

Audio, video, and images are compelling forms of content that the majority of apps use. As media capture rates increase and content moves from standard definition to high definition the amount of resources need to store, decode, and play back this content increases. The XAML framework builds on the latest features added to the Windows 8 media engines so apps get these improvements for free. Here we explain some additional tricks that allow you to get the most out media in your Windows Store app using C++, C#, or Visual Basic for Windows 8.

Release media streams

Media files are some of the most common and expensive resources apps typically use. Because media file resources can greatly increase the size of your app's memory footprint, you must remember to release the handle to media as soon as the app is finished using it.

For example, if your app working with a RandomAccessStream or an IInputStream object, be sure to call the close method on the object when your app has finished using it, to release the underlying object.

Video

Display full screen video playback when possible

In Windows 8.1 and later, always use the IsFullWindow property on the MediaElement to enable and disable full window rendering. This insures system level optimizations are used during media playback.

The XAML framework can optimize the display of video content when it is the only thing being rendered, resulting in an experience that uses less power and yields higher frame rates. For most efficient media playback set the size of a MediaElement to be the width and height of the screen and don’t display other XAML elements

There are legitimate reasons to overlay XAML elements on a MediaElement that takes up the full width and height of the screen, for example closed captions or momentary transport controls. Making sure to hide these elements (eg. setting Visibility=”Collapsed”) when they are not needed pops media playback back into its most efficient sate.

Display deactivation and conserving power

To prevent the display from be deactivating when user action is no longer detected, such as when an app is playing video, you can call DisplayRequest.RequestActive.

To conserve power and battery life, you should call DisplayRequest.RequestRelease to release the display request as soon as it is no longer required.

Here are some situations when you should release the display request:

Video playback is paused, for example by user action, buffering, or adjustment due to limited bandwidth.
Playback stops. For example, the video is done playing or the presentation is over.
A playback error has occurred. For example, network connectivity issues or a corrupted file.

Put other elements to the side of embedded video

Often apps offer an embedded view where video is played within a page. Now you obviously lost the full screen optimization because the MediaElement is not the size of the page and there are other XAML objects drawn. Beware of unintentionally entering this mode by drawing a border around a MediaElement.

Don’t draw XAML elements on top of video when it’s in embedded mode. If you do, the framework is forced to do a little extra work to compose the scene. Placing transport controls below an embedded media element instead of on top of the video is a good example of optimizing for this situation. In this image, the red bar indicates a set of transport controls (play, pause, stop, etc.).

Don’t place these controls on top of media that is not full screen. Instead place the transport controls somewhere outside of the area where the media is being rendered. In the next image, the controls are placed below the media.

Delay setting the source for a MediaElement

Media engines are expensive objects and the XAML framework delays loading dlls and creating large objects as long as possible. The MediaElement is forced to do this work after its source is set via the Source property or the SetSource method. Setting these when the user is really ready to play media delays the majority of the cost associated with the MediaElement as long as possible.

Set MediaElement.PosterSource

Setting MediaElement.PosterSource enables XAML to release some GPU resources that would have otherwise been used. This API allows an app to use as little memory as possible.

Improve scrubbing

Scrubbing is always a tough task for media platforms to make really responsive. Generally people accomplish this by changing the value of a Slider. Here are a couple tips on how to make this as efficient as possible:

Either bind the value of a Slider to MediaElement.Position or update it based on a timer. Don't do both. If you choose the latter, make sure to use a reasonable update frequency for your timer. The XAML framework only updates MediaElement.Position only every 250 milliseconds during playback.
The size of the step frequency on the Slider must scale with the length of the video.
Subscribe to the PointerPressed, PointerMoved, PointerReleased events on the slider to set the MediaElement.PlaybackRate property to 0 when the user drags the thumb of the slider.
In the PointerReleased event handler, manually set the media position to the slider position value to achieve optimal thumb snapping while scrubbing.

Match video resolution with device resolution

Decoding video takes a lot of memory and GPU cycles, so choose a video format close to the resolution it will be displayed at. There is no point in using the resources to decode 1080 video if it’s going to get scaled down to a much smaller size. Many apps don’t have the same video encoded at different resolutions; but if it is available, use an encoding that is close to the resolution at which it will be displayed.

Choose recommended formats

Media format selection can be a sensitive topic and is often driven by business decisions. From a Windows 8 performance perspective, we recommend H.264 video as the primary video format and AAC and MP3 as the preferred audio formats. For local file playback, MP4 is the preferred file container for video content. H.264 decoding is accelerated through most recent graphics hardware. Also, although hardware acceleration for VC-1 decoding is broadly available, for a large set of graphics hardware on the market, the acceleration is limited in many cases to a partial acceleration level (or IDCT level), rather than a full-steam level hardware offload (i.e. VLD mode).

If you have full control of the video content generation process, you must figure out how to keep a good balance between compression efficiency and GOP structure. Relatively smaller GOP size with B pictures can increase the performance in seeking or trick modes.

When including short, low-latency audio effects, for example in games, use WAV files with uncompressed PCM data to reduce processing overhead that is typical for compressed audio formats.

Images

Scale images to the appropriate size

Images are now captured at very high resolutions. This is great for capturing vibrant color and small details, but requires that apps use more CPU decoding the image data and more memory after it’s loaded from disk. There’s no sense decoding and saving a high resolution image in memory if it will never be displayed at its native size. Instead, reduces CPU usage and memory footprint by creating a lower resolution version of the image that is close to the display size. Use the DecodePixelWidth and DecodePixelHeight properties to produce image objects that are close to the size to display in the app.

Don't do this.

<Image Source="ms-appx:///Assets/highresCar.jpg" 
       Width="300" Height="200"/>    <!-- BAD CODE DO NOT USE.-->

Instead, do this

<Image>
    <Image.Source>
    <BitmapImage UriSource="ms-appx:///Assets/highresCar.jpg" 
                 DecodePixelWidth="300" DecodePixelHeight="200"/>
    </Image.Source>
</Image>

Use GetThumbnailAsync for thumbnails

One instance of scaling images is creating thumbnails. Although you could use DecodePixelWidth and DecodePixelHeight to provide small versions of images, the Windows Runtime provides even more efficient APIs for retrieving thumbnails. GetThu mbnailAsync provides the thumbnails for images that have the file system already cached. This provides even better performance than the XAML APIs because the image doesn’t need to be opened or decoded.

FileOpenPicker picker = new FileOpenPicker();
picker.FileTypeFilter.Add(".bmp");
picker.FileTypeFilter.Add(".jpg");
picker.FileTypeFilter.Add(".jpeg");
picker.FileTypeFilter.Add(".png");
picker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;

StorageFile file = await picker.PickSingleFileAsync();

StorageItemThumbnail fileThumbnail = await file.GetThumbnailAsync(ThumbnailMode.SingleItem, 64);

BitmapImage bmp = new BitmapImage();
bmp.SetSource(fileThumbnail);

Image img = new Image();
img.Source = bmp;

Dim picker As New FileOpenPicker()
picker.FileTypeFilter.Add(".bmp")
picker.FileTypeFilter.Add(".jpg")
picker.FileTypeFilter.Add(".jpeg")
picker.FileTypeFilter.Add(".png")
picker.SuggestedStartLocation = PickerLocationId.PicturesLibrary

Dim file As StorageFile = Await picker.PickSingleFileAsync()

Dim fileThumbnail As StorageItemThumbnail = Await file.GetThumbnailAsync(ThumbnailMode.SingleItem, 64)

Dim bmp As New BitmapImage()
bmp.SetSource(fileThumbnail)

Dim img As New Image()
img.Source = bmp

Decode images once

To prevent images from being decoded more than once, assign the Image.Source property from an Uri rather than using memory streams. The XAML framework can associate the same Uri in multiple places with one decoded image, but it cannot do the same for multiple memory streams that contain the same data and creates a different decoded image for each memory stream.

Audio

Hardware audio offloading

For hardware audio offload to be automatically applied, MediaElement.AudioCategory must be set to ForegroundOnlyMedia or BackgroundCapableMedia. Hardware audio offload optimizes audio rendering which can improve functionality and battery life.

Partager via