• Article

October 2013

Volume 28 Number 10

Windows Phone - Programming the Nokia Sketch Effect in Windows Phone 8

By Srikar Doddi

Nokia recently released the Nokia Imaging SDK in beta to allow Windows Phone 8 developers to create advanced imaging experiences for Nokia Lumia smartphones.

The Nokia Imaging SDK includes a library for manipulating images captured and stored by Windows Phone devices. Its features include decoding and encoding JPEG images, applying filters and effects, cropping, rotating, and resizing. The Nokia Imaging SDK provides more than 50 premade filters and effects. You can not only apply effects such as sepia, cartoon, sketch, and so on, but also apply auto-enhance, brightness control, hue, saturation, and many more. The SDK has been specifically developed for mobile imaging, with speed and memory performance as key drivers.

In this article, I’ll demonstrate the use of the sketch effect through an example app called Paper Photo. The sketch effect is applied to the viewfinder stream in real time and provides users the ability to capture images and save them to the phone’s camera roll. I developed this app using Visual Studio 2012, the Nokia Imaging SDK and the Windows Phone 8 SDK. I used a Nokia Lumia 920 smartphone to test it.

Nokia Imaging SDK Basics

The Nokia Imaging SDK was created by Nokia to give you full access to a powerful library of image-manipulation tools that help you create great imaging apps quickly and easily. As mentioned, this SDK was designed with high performance and low-memory usage in mind, which is important because Microsoft and Nokia are working to create next-generation devices such as the Nokia Lumia 1020. The SDK is already used in Nokia’s own imaging application, Creative Studio. The SDK and the library are available free of charge (see the license agreement at bit.ly/130tVHJ), and the SDK currently supports only Windows Phone 8 apps.

The Nokia Imaging SDK offers the following features:

  • A simple-to-use API that’s available both from managed code as well as native code. This means the SDK is provided as a Windows Phone Runtime library and you can call the methods of the library from C#, Visual Basic or C++.The API comes with a range of classes and methods for various imaging tasks.
  • RAJPEG technology to access image data without decoding a whole JPEG image first, which allows for blazingly fast previews, application of effects, cropping and rotation of high-resolution images.
  • More than 50 filters, effects and enhancements. Some of these enhancements allow you to programmatically adjust RGB levels, hue, saturation and brightness. You can also create custom effects and filters if needed. In addition to these filters and effects, the SDK also allows for cropping, rotating and resizing with unlimited undo functionality.
  • Several fully featured sample apps allow you to explore the source code and discover the features of the SDK. The examples cover a range of features such as real-time manipulation of filter properties and application of several filter layers to photos.
  • Rich documentation with quick-start guides, example projects, API reference guides and several documents that provide an overview of some of the core concepts needed to create imaging apps.

Getting Started

At a high level, I used the following APIs from the Nokia Imaging SDK for my sample app:

  • Nokia.Graphics
  • Nokia.Graphics.Imaging
  • Nokia.InteropServices.WindowsRuntime

The Nokia.Graphics.Imaging API contains the core functionality of the Nokia Imaging SDK, which includes all the image filters, image effects, JPEG encoder and decoder. The Nokia.InteropServices.WindowsRuntime library is used internally and is a required library that needs to be referenced in your project. It contains a class called BufferFactory that’s used for creating an instance of IBuffer. This SDK can be installed using the NuGet package manager in Visual Studio. You can find the package by searching for NokiaImagingSDK.

The object graph of this app basically consists of three key classes, shown in Figure 1. The MainPage is your typical phone app page implemented by a XAML file and a C# counterpart. This MainPage class implements the app UI, which includes the MediaElement that displays the camera viewfinder with the sketch effect. The MainPage class also owns the instances of the two other key classes: CameraStreamSource and NokiaSketchEffect. The CameraStreamSource, derived from MediaStreamSource, provides the camera data, and the NokiaSketchEffect implements the sketch effect. The CameraStreamSource is a class implemented by Nokia and is provided to developers through the company’s app samples outside of the Nokia Imaging SDK.

The Class Diagram for the Paper Photo Project
Figure 1 The Class Diagram for the Paper Photo Project

The object graph translates to the code organization shown in Figure 2.

Visual Studio Solution Structure
Figure 2 Visual Studio Solution Structure

Defining Your UI

The UI of this app is straightforward, as shown in Figure 3. The main page displays the image from a viewfinder and comes with an application bar that has a single button to capture the photo with the sketch mode effect.

The Paper Photo UI
Figure 3 The Paper Photo UI

Figure 4 shows the XAML markup for the main page using the Grid element to define the container used for showing the viewfinder with the sketch effect applied.

Figure 4 The XAML Markup for the Main Page

<Grid x:Name="LayoutRoot" Background="Transparent">
  <Grid.RowDefinitions>
    <RowDefinition Height="Auto"/>
    <RowDefinition Height="*"/>
  </Grid.RowDefinitions>
  <Grid Grid.Row="1" Margin="8">
    <Grid x:Name="MediaElementContainer" Margin="0"></Grid>
      <StackPanel
        x:Name="TitlePanel"
        Grid.Row="0"
        Margin="12,17,0,28">
        <TextBlock
          Text="{Binding Path=LocalizedResources.ApplicationTitle,
          Source={StaticResource LocalizedStrings}}"
          Style="{StaticResource PhoneTextNormalStyle}"
          Margin="12,0"/>
      </StackPanel>           
    </Grid>
  </Grid>

The application bar is built using the markup shown in Figure 5. As you can see, it defines a capture button and an About menu item. The capture button is associated with an event handler to handle the click event.

Figure 5 The Application Bar Markup

<phone:PhoneApplicationPage.ApplicationBar>
  <shell:ApplicationBar Opacity="0.4">
    <shell:ApplicationBarIconButton x:Name="CaptureButton"
      Text="Capture" IconUri="Assets/capture-button-icon.png"
      Click="CaptureButton_Click" IsEnabled="True" />
    <shell:ApplicationBar.MenuItems>
      <shell:ApplicationBarMenuItem
        Click="OnAboutPageButtonClicked" Text="about" />
    </shell:ApplicationBar.MenuItems>
  </shell:ApplicationBar>
</phone:PhoneApplicationPage.ApplicationBar>

Windows Phone uses a capabilities-driven security model. This model allows users to opt in to allow certain functionality to be activated. In my demo app, the following capabilities need to be enabled for users:

  • ID_CAP_ISV_CAMERA: This provides access to the rear camera (primary) or the front-facing camera.
  • ID_CAP_MEDIALIB_PHOTO: This provides read-only access to photos in the media library. It also gives an app the ability to save photos in the camera roll. I’ll demonstrate this capability later in the article to save a photo to the camera roll after applying the sketch effect.

In terms of the hardware requirements, the app requires a rear-facing camera to function properly. The ID_REQ_REARCAMERA option is selected to prevent the app from installing on a phone without a rear camera.

Applying a Real-Time Sketch Effect to the Camera Viewfinder

The NokiaSketchEffect class is responsible for applying the sketch effect in real time to the camera viewfinder. Note the use of SketchMode.Gray to get the desired effect in my app. The Get­SampleAsync function in the CameraStreamSource class is responsible for processing the camera buffer using the sketch effect, and it provides the media element with the buffer. The GetSampleAsync method uses the Nokia­SketchEffect.GetNewFrameAndApplyEffect method to get the modified camera buffer. The code in Figure 6 shows how it’s implemented.

Figure 6 Getting the Modified Camera Buffer

public async Task GetNewFrameAndApplyEffect(IBuffer processedBuffer)
{
  if (captureDevice == null)
  {
    return;
  }
  captureDevice.GetPreviewBufferArgb(cameraBitmap.Pixels);
  Bitmap outputBtm = new Bitmap(
    outputBufferSize,
    ColorMode.Bgra8888,
    (uint)outputBufferSize.Width * 4,
    processedBuffer);
  EditingSession session = 
    new EditingSession(cameraBitmap.AsBitmap());
  session.AddFilter(FilterFactory.CreateSketchFilter(SketchMode.Gray));
  await session.RenderToBitmapAsync(outputBtm);
}

The captureDevice in the function is a private variable of the Photo­CaptureDevice class found in Windows.Phone.Media.Capture. The RenderToBitmapAsync function is provided by the Nokia Imaging SDK to support the asynchronous render of EditingSession to a bitmap. The EditingSession is an important class provided by the Nokia Imaging SDK to represent an image-processing editing session. The EditingSession object is the core of this SDK. In a typical workflow, the EditingSession is created from an image, filters or effects are added to this session, the EditingSession is then rendered to a bitmap or into a memory buffer, and finally the EditingSession is closed. The code in Figure 6 shows how a new bitmap associated with the processedBuffer argument is created, the sketch effect is applied to it and then it’s rendered into the new buffer. Finally, the CreateSketchFilter method produces the look of the sketched image. This function takes SketchMode—an enumeration—as an argument to pass the modes for the sketch filter. The two modes available are gray for sketch in grayscale and color for sketch in color. As you can see, I used SketchMode.Gray in the Figure 6 code. This allows the Paper Photo app to produce images in grayscale.

Capturing and Saving the Picture as a JPEG Image to the Camera Roll

So far I’ve shown how to apply the sketch effect in real time to the viewfinder. Now I’ll look at capturing a picture from the viewfinder and then saving it to the camera roll on the device. The capture function first initiates autofocus and then captures a photo. Besides handling the capture functionality from the application bar, you can also handle the capture initiated by the hardware device trigger.

The code shown in Figure 7 enables capture through the camera button. More specifically, the function enables or disables the hardware shutter release function. The CameraButtons class provides the events that are triggered by the device shutter buttons. One such event is the ShutterKeyHalfPressed event, triggered when the hardware shutter button is pressed and held for approximately 800 milliseconds. Another event is the ShutterKeyPressed event. This event occurs when the hardware shutter button receives a full press. The SetCameraButtonsEnabled function also handles the removal of these event handlers to help release the memory related to the camera.

Figure 7 Enabling Capture Through the Camera Button

private void SetCameraButtonsEnabled(bool enabled)
{
  if (enabled)
  {
    Microsoft.Devices.CameraButtons.ShutterKeyHalfPressed
      += ShutterKeyHalfPressed;
    Microsoft.Devices.CameraButtons.ShutterKeyPressed
      += ShutterKeyPressed;
  }
  else
  {
    Microsoft.Devices.CameraButtons.ShutterKeyHalfPressed
      -= ShutterKeyHalfPressed;
    Microsoft.Devices.CameraButtons.ShutterKeyPressed
      -= ShutterKeyPressed;
  }
}

Now I’ll explain the capture process in detail. In the code shown in Figure 8, the camera object represents the photo capture device, the await camera.FocusAsync function initiates the autofocus and the StartCaptureAsync method captures a frame. But before you can capture the frame, you need to prepare the capture sequence. You do this by calling PrepareCaptureSequenceAsync. You’re also creating a capture sequence object with one frame, as you can see from camera.CreateCaptureSequence(1). The value 1 indicates the number of frames that will be captured immediately after you initiate the capture. Finally, you also specify camera properties and photo settings using the KnownCameraPhotoProperties method. The LockedAutoFocusParameters is set to None, indicating that focus, exposure and white balance settings will adjust automatically prior to capture.

Figure 8 The Capture Process

private async System.Threading.Tasks.Task Capture()
{
  try
  {
    await camera.FocusAsync();
    MemoryStream imageStream = new MemoryStream();
    imageStream.Seek(0, SeekOrigin.Begin);
    CameraCaptureSequence sequence = camera.CreateCaptureSequence(1);
    sequence.Frames[0].CaptureStream = imageStream.AsOutputStream();
    await camera.PrepareCaptureSequenceAsync(sequence);
    await sequence.StartCaptureAsync();
    camera.SetProperty(
      KnownCameraPhotoProperties.LockedAutoFocusParameters,
      AutoFocusParameters.None);
    ...
}

Finally, the following code shows how to save the captured photo to the camera roll:

MediaLibrary library = new MediaLibrary();
EditingSession session =
  new EditingSession(imageStream.GetWindowsRuntimeBuffer());
using (session)
{
  session.AddFilter(FilterFactory.CreateSketchFilter(SketchMode.Gray));
  IBuffer data = await session.RenderToJpegAsync();
  library.SavePictureToCameraRoll(FileNamePrefix
    + DateTime.Now.ToString() + ".jpg",
    data.AsStream());
}

The MediaLibrary class is provided by the Microsoft.XNA.Framework.Media API and is used to save photos to the phone’s media library. The Microsoft.XNA.Framework.Media namespace contains classes to enumerate, play, and view songs, albums, playlists, and pictures. You then use the EditingSession class from the Nokia Imaging SDK to create a session from the compressed image buffer and then apply the sketch effect filter before saving the picture to the camera roll using the SavePictureToCameraRoll function.

Lens Picker Integration

In Windows Phone 8, you can create a rich media lens that opens from the built-in camera app and launches your app right into the viewfinder experience. In order to integrate with the lens experience, the app needs to register for the Camera_Capture_App extension. This extension declares to the OS that the Paper Photo app can display a viewfinder when it’s launched from the lens picker. Extensions are specified in the WMAppManifest.xml file. You need to open this file with the text editor: Right-click on the file and choose “Open With… | Text Editor.” Just after the Tokens element, inside the Extensions element, the lens extension is specified with the following Extension element:

<Extension ExtensionName="Camera_Capture_App"
  ConsumerID="{5B04B775-356B-4AA0-AAF8-6491FFEA5631}"
  TaskID="_default" />

Now your app is complete with lens integration and can save pictures right into the camera roll on the device. The image in Figure 9 was taken with the built-in camera app, and the image in Figure 10 was taken with the Paper Photo app using the sketch effect.

A Regular Image Taken with the Built-in Camera App
Figure 9 A Regular Image Taken with the Built-in Camera App

An Image Taken Using the Paper Photo App Using the Sketch Effect
Figure 10 An Image Taken Using the Paper Photo App Using the Sketch Effect

Add Your Own Features

In this article, I explored photo capture APIs from Microsoft and the brand-new Nokia Imaging SDK to create a rich imaging experience. I applied the sketch effect to the viewfinder in real time and then captured the photo and saved it on the device camera roll. As you can see, there are tons of features that you can now use thanks to the Nokia Imaging SDK, which makes it easy to build apps such as this one. A few minor details were omitted for the sake of brevity, but you can refer to the code download (github.com/Srikar-Doddi/PaperPhoto) for a complete understanding. This is just one example of using the Nokia Imaging SDK, but there are lots of other features you can add and the possibilities are unlimited. I hope you use this code as a starting point and add more features to this app.

Srikar Doddi is the executive director of engineering at Kaplan Test Prep. You can find his writings on Medium at medium.com/@SrikarDoddi. He is also the creator of the Simplist, DateTileScheduler and Paper Photo apps for Windows Phone 8 and the Prompter and Dabble apps for Windows 8. You can reach Doddi via e-mail at srikar.doddi@gmail.com.

Thanks to the following technical expert for reviewing this article: Lance McCarthy (Nokia)
Lance McCarthy is a Nokia ambassador and Telerik XAML Support Specialist (ext-lance.mccarthy@nokia.com)