Filter Effects for Windows Phone 8

Filter Effects is an example app demonstrating the use of the different filters of Lumia Imaging SDK with camera photos. This example app uses the camera and displays the viewfinder for taking a picture. The taken photo is then processed with the predefined filters. The processed image can be saved in JPEG format into the camera roll album. You can also select an existing photo and apply an effect to it.

Compatibility

• Compatible with Windows Phone 8.
• Tested with Nokia Lumia 920 and Nokia Lumia 620.
• Developed with Visual Studio 2013 Express for Windows Phone 8.
• Compiling the project requires the Lumia Imaging SDK.

Architecture overview

The example architecture is kept simple; the main functionality is implemented in two classes: MainPage and PreviewPage. However, for easy addition of new filters, a simple, abstract base class for each filter is implemented: AbstractFilter. DataContext class implements the singleton pattern and holds the memory streams of camera data. FilterPropertiesControl class is a custom XAML control that is populated by a concrete filter class. The control is composed of other controls which can be used to manipulate the properties of the filter associated with the control.

Capturing a photo

The implementation for capturing a photo is located in the MainPage class (see MainPage.xaml.cs). The class for capturing photos, provided by the Windows Phone framework, is PhotoCaptureDevice. To initialize the class instance, you must provide the sensor location, defining which physical camera to use (front or back), and the resolution for the images to capture. Initializing the camera may take a while, so it must be done asynchronously to avoid blocking the user interface (UI).

private PhotoCaptureDevice _photoCaptureDevice = null; // Class member
// ...
private async Task InitializeCamera(CameraSensorLocation sensorLocation)
{
    // ...

    PhotoCaptureDevice device = await PhotoCaptureDevice.OpenAsync(sensorLocation, initialResolution);
    await device.SetPreviewResolutionAsync(previewResolution);
    await device.SetCaptureResolutionAsync(captureResolution);
    _photoCaptureDevice = device;
}

Capturing a photo is implemented in MainPage.Capture.

private async Task Capture()
{
    // ...

    DataContext dataContext = FilterEffects.DataContext.Singleton;

    // Reset the streams
    dataContext.ImageStream.Seek(0, SeekOrigin.Begin);
    dataContext.ThumbStream.Seek(0, SeekOrigin.Begin);

    CameraCaptureSequence sequence = _photoCaptureDevice.CreateCaptureSequence(1);
    sequence.Frames[0].CaptureStream = dataContext.ImageStream.AsOutputStream();
    sequence.Frames[0].ThumbnailStream = dataContext.ThumbStream.AsOutputStream();

    await _photoCaptureDevice.PrepareCaptureSequenceAsync(sequence);
    await sequence.StartCaptureAsync();

    // ...
}

Note that you need ID_CAP_ISV_CAMERA capability to access camera device.

Processing the image data

The image processing is managed (but not implemented) by the PreviewPage class.

private async void CreatePreviewImages() 
{
    DataContext dataContext = FilterEffects.DataContext.Singleton;
    // ...
    foreach (AbstractFilter filter in _filters)
    {
        filter.Buffer = dataContext.ImageStream.GetWindowsRuntimeBuffer();
        filter.Apply();
    }
}

Most of the image processing logic is implemented in the abstract base class, AbstractFilter. The derived classes only define the filters to use and their properties; each filter implements the abstract method SetFilters. Here is an example of how it is implemented in SixthGearFilter.cs.

public class SixthGearFilter : AbstractFilter 
{
    // ...

    protected LomoFilter _lomoFilter; // Class member

    // ...

    protected override void SetFilters(FilterEffect effect)
    {
        effect.Filters = new List<IFilter>() { _lomoFilter };
    }

    // ...
}

Calling AbstractFilter.Apply will schedule an image processing procedure. If there is no queue, the buffer is processed right away (in AbstractFilter.Render()):

public abstract class AbstractFilter : IDisposable 
{
    // ...

    // Members
    private FilterEffect _effect; // The filters are set by the derived class in SetFilters()
    private WriteableBitmap _previewBitmap; // This is bound to the preview image shown on the screen
    private WriteableBitmap _tmpBitmap;

    // ...

    protected async void Render()
    {
        // ...

        // Render the filters first to the temporary bitmap and copy the changes then to the preview bitmap
        using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(_effect, _tmpBitmap))
        {
            await renderer.RenderAsync();
        }
        _tmpBitmap.Pixels.CopyTo(_previewBitmap.Pixels, 0);
        _previewBitmap.Invalidate(); // Force a redraw

        // ...
    }

    // ...
}

Saving the processed image

The image saving is implemented in the PreviewPage.SaveButton_Click method.

private async void SaveButton_Click(object sender, EventArgs e) 
{
 
    // ...

    AbstractFilter filter = _filters[selectedIndex];
    IBuffer buffer = await filter.RenderJpegAsync(dataContext.ImageStream.GetWindowsRuntimeBuffer());
    // ...

}

Note that a separate helper method, RenderJpegAsync, is implemented by the AbstractFilter class.

public async Task<IBuffer> RenderJpegAsync(IBuffer buffer) 
{
    using (BufferImageSource source = new BufferImageSource(buffer))
    using (JpegRenderer renderer = new JpegRenderer(_effect))
    {
        return await renderer.RenderAsync();
    }
}

After the image data has been processed and the filtered image data is rendered to the buffer, it can be saved to the camera roll in media library (in PreviewPage.SaveButton_Click).

// ...

    using (MediaLibrary library = new MediaLibrary())
    {
        library.SavePictureToCameraRoll(FileNamePrefix + DateTime.Now.ToString() + ".jpg", buffer.AsStream());
    }

// ...

Note that you need ID_CAP_MEDIALIB_PHOTO capability to store images into the media library.

Adding a new filter

You can modify the existing filter or you can easily add a new one. For a new filter, just implement the abstract base class. The only method you need to implement is SetFilters.

Tip: For an easy start, copy the source of any of the existing filters. To add the new filter to the collection, just add a new line to the toCreateFilters method of the PreviewPage class.

private void CreateFilters()
{
    _filters = new List<AbstractFilter>();
    _filters.Add(new OriginalImageFilter()); // This is for the original image and has no effects
    _filters.Add(new SixthGearFilter());
    _filters.Add(new SadHipsterFilter());
    _filters.Add(new EightiesPopSongFilter());
    _filters.Add(new CartoonFilter());
    _filters.Add(new MyNewFilter()); // <-- This is the line to add
}

Creating a custom control to modify the filter properties

Different filters have different properties:

The FilterPropertiesControl class, implemented in FilterPropertiesControl.xaml, and FilterPropertiesControl.xaml.cs, is placeholder for custom controls used to manipulate the filter properties. Since different filters have different kind of parameters, the controls are filter-specific. Therefore, a concrete filter class, derived from AbstractFilter, has to know how to populate the controls. Below is an example of how the controls are populated for the SixthGearFilter class.

public override bool AttachControl(FilterPropertiesControl control)
{
    _control = control;
    Grid grid = new Grid();
    int rowIndex = 0;

    TextBlock brightnessText = new TextBlock();
    brightnessText.Text = AppResources.Brightness;
    Grid.SetRow(brightnessText, rowIndex++);

    Slider brightnessSlider = new Slider();
    brightnessSlider.Minimum = 0.0;
    brightnessSlider.Maximum = 1.0;
    brightnessSlider.Value = _lomoFilter.brightness;
    brightnessSlider.ValueChanged += brightnessSlider_ValueChanged;
    Grid.SetRow(brightnessSlider, rowIndex++);

    TextBlock saturationText = new TextBlock();
    saturationText.Text = AppResources.Saturation;
    Grid.SetRow(saturationText, rowIndex++);

    Slider saturationSlider = new Slider();
    saturationSlider.Minimum = 0.0;
    saturationSlider.Maximum = 1.0;
    saturationSlider.Value = _lomoFilter.Saturation;
    saturationSlider.ValueChanged += saturationSlider_ValueChanged;
    Grid.SetRow(saturationSlider, rowIndex++);

    TextBlock lomoVignettingText = new TextBlock();
    lomoVignettingText.Text = AppResources.LomoVignetting;
    Grid.SetRow(lomoVignettingText, rowIndex++);

    RadioButton lowRadioButton = new RadioButton();
    lowRadioButton.GroupName = _lomoVignettingGroup;
    TextBlock textBlock = new TextBlock();
    textBlock.Text = AppResources.Low;
    lowRadioButton.Content = textBlock;
    lowRadioButton.Checked += lowRadioButton_Checked;
    Grid.SetRow(lowRadioButton, rowIndex++);

    RadioButton medRadioButton = new RadioButton();
    medRadioButton.GroupName = _lomoVignettingGroup;
    textBlock = new TextBlock();
    textBlock.Text = AppResources.Medium;
    medRadioButton.Content = textBlock;
    medRadioButton.Checked += medRadioButton_Checked;
    Grid.SetRow(medRadioButton, rowIndex++);

    RadioButton highRadioButton = new RadioButton();
    highRadioButton.GroupName = _lomoVignettingGroup;
    textBlock = new TextBlock();
    textBlock.Text = AppResources.High;
    highRadioButton.Content = textBlock;
    highRadioButton.Checked += highRadioButton_Checked;
    Grid.SetRow(highRadioButton, rowIndex++);

    switch (_lomoFilter.LomoVignetting)
    {
        case LomoVignetting.Low: lowRadioButton.IsChecked = true; break;
        case LomoVignetting.Medium: medRadioButton.IsChecked = true; break;
        case LomoVignetting.High: highRadioButton.IsChecked = true; break;
    }

    for (int i = 0; i < rowIndex; ++i)
    {
        RowDefinition rd = new RowDefinition();
        grid.RowDefinitions.Add(rd);
    }

    grid.Children.Add(brightnessText);
    grid.Children.Add(brightnessSlider);
    grid.Children.Add(saturationText);
    grid.Children.Add(saturationSlider);
    grid.Children.Add(lomoVignettingText);
    grid.Children.Add(lowRadioButton);
    grid.Children.Add(medRadioButton);
    grid.Children.Add(highRadioButton);

     control.ControlsContainer.Children.Add(grid);
            
    return true;
}

Modifying filter properties on the fly

When you want to modify the filter properties so that the changes can be previewed instantaneously while maintaining smooth user experience, you are faced with two problems:

1. If the filter property value changes when the rendering process is ongoing, InvalidOperationException is thrown.

2. Rendering to a bitmap which is already being used for rendering may lead to unexpected results.

One might think that catching the exception thrown in the first problem would suffice, but then the user might start wondering why the changes he wanted to make did not have an effect on the image. In addition to the poor user experience (UX), you would still have to deal with the second problem.

To solve both problems, a simple state machine can be implemented. In AbstractFilter class there are three defined states and a declared property, State, to keep track of the current state.

public abstract class AbstractFilter : IDisposable 
{
    protected enum States
    {
        Wait = 0,
        Apply,
        Schedule
    };

    private States _state = States.Wait;
    protected States State
    {
        get
        {
            return _state;
        }
        set
        {
            if (_state != value)
            {
                _state = value;
            }
        }
    }

    // ...

} 

The transitions are as follows:

• Wait to Apply until a request for processing is received.
• Apply to Schedule when a new request is received while processing the previous request.
• Schedule to Apply when the previous processing is complete and a pending request is taken to processing.
• Apply to Wait when the previous processing is complete and no request is pending.

The state is managed by two methods in AbstractFilter class: Apply, which is public, and Render, which is protected.

public void Apply() 
{
    switch (State)
    {
        case States.Wait: // State machine transition: Wait -> Apply
            State = States.Apply;
            Render(); // Apply the filter
            break;
        case States.Apply: // State machine transition: Apply -> Schedule
            State = States.Schedule;
            break;
        default:
            // Do nothing
            break;
    }
}

As you can see, Render, which does the actual processing, will only be called when the current state is Wait.

protected async void Render() 
{
    // Apply the pending changes to the filter(s)
    foreach (var change in _changes)
    {
        change();
    }

    _changes.Clear();

    // Render the filters first to the temporary bitmap and copy the changes then to the preview bitmap
    using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(_effect, _tmpBitmap))
    {
        await renderer.RenderAsync();
    }
    _tmpBitmap.Pixels.CopyTo(_previewBitmap.Pixels, 0);
    _previewBitmap.Invalidate(); // Force a redraw

    switch (State)
    {
        case States.Apply: // State machine transition : Apply -> Wait
            State = States.Wait;
            break;
        case States.Schedule: // State machine transition: Schedule -> Apply
            State = States.Apply;
            Render(); // Apply the filter
            break;
        default:
            // Do nothing
            break;
    }
}

Note: Some of the error handling is omitted from the snippet above. Also pay attention to the part in the beginning of the method; namely this part.

// Apply the pending changes to the filter(s)
foreach (var change in _changes)
{
    change();
}

_changes.Clear();

The type of _changes is List<Action> and it is a protected member variable of the class AbstractFilter. The list is populated by the derived classes; every single change to filter properties is added to the list before a new request to process the image is made. This is how we can generalize the property change regardless of the filter or the type of the property. Here, for example, is the code used when the user adjusts the Brightness property of the lomo filter in SixthGearFilter.

protected void brightnessSlider_ValueChanged(object sender, System.Windows.RoutedPropertyChangedEventArgs<double> e) 
{
    _changes.Add(() => { _lomoFilter.Brightness = 1.0 - e.NewValue; });
    Apply();
    _control.NotifyManipulated(); 
}

See also

• Camera Explorer example for advanced camera implementation.

Downloads

Filter Effects project

filter-effects-master.zip

This example application is hosted in GitHub, in a single project for Windows Phone and Windows, where you can check the latest activities, report issues, browse source, ask questions or even contribute yourself to the project.

Note: In order to deploy the application binary file yourself to a Windows Phone device, not via store, you must be a registered developer and your phone must be registered for development. For more details, see the deploying and running apps on a Windows Phone device (MSDN).