Hope you can come along!

First events are coming up fast!:

• Webinar 1: Thursday 21^st August, 10:00 – 11:00
  Registration: https://aka.ms/winappswebinar1
• Webinar 2: Thursday 21^st August 11:00 – 12:00
  Registration: https://aka.ms/winappswebinar2
• Dev camp: Date to be announced soon…
  Registration: *tbd*
• Publish hackathon: Saturday September 20^th 9:00 – 21:00
  Registration: https://aka.ms/winappspublishhack

Come along and join us – and if not this quarter, remember we are repeating these events throughout the year so you can join us when you can!

Opening databases shipped as content

You can include a prepopulated SQLite database in your Windows 8.x or Windows Phone app package. When the user installs the app, the database file ends up in the App Install folder where you can open it from your code – but only in a read-only fashion:

public static async void LoadDatabase()
{
    // Get a reference to the SQLite database
    db = new SQLiteWinRT.Database(
        Windows.ApplicationModel.Package.Current.InstalledLocation, 
        "customers.sqlite");

    await db.OpenAsync(SQLiteWinRT.SqliteOpenMode.OpenRead);

    // PRAGMA temp_files=2 causes temporary files to be created in memory
    // rather than in a physicalfile in the same folder as the database
    // Note that temp files only really get used if you are going to do 
    // complicated JOINs etc
    await db.ExecuteStatementAsync("PRAGMA temp_files=2");
}

If you want to have read-write access to the database, you must copy it from the install folder to the local folder first.

async Task CopyDatabaseAsync()
{
    // Has the database been copied already?
    try
    {
        await ApplicationData.Current.LocalFolder.GetFileAsync("customers.sqlite");
        // No exception? It exists...
        return;
    }
    catch (System.IO.FileNotFoundException)
    {   // Expected response - no-op 
    }

    StorageFile dbfile = await StorageFile.GetFileFromApplicationUriAsync(
        new Uri("ms-appx:///customers.sqlite"));
    await dbfile.CopyAsync(ApplicationData.Current.LocalFolder);
}

Opening a Database on an SD Card

A database file that you open from an SD Card is related to the previous case in that it is read-only. You cannot write to any file on an SD Card from any app in Windows Phone 8.

I haven’t researched it thoroughly, but you should be able to open a database file on removable storage in a Windows 8.x Store app. You will have to request the Removable Storage capability in the app manifest, and declare the file extensions you want to access in the app manifest as well. Then you should be able to get a StorageFile reference to the database file using techniques similar to those discussed here: http://lunarfrog.com/blog/2012/10/27/external-storage-devices/   . When I’ve tried it, I’ll update this post!

What I have tried though, is doing this on a Windows Phone 8 device that has an SD card slot, such as the Nokia Lumia 820. As with Windows 8, you have to declare the file extensions you want to access in the manifest. For example, add the following after the closing </Tokens> tag:

    <Extensions>
      <FileTypeAssociation Name="sqlite" TaskID="_default" NavUriFragment="fileToken=%s">
        <SupportedFileTypes>
          <FileType ContentType="application/sqlite">.sqlite</FileType>
        </SupportedFileTypes>
      </FileTypeAssociation>
    </Extensions>

Then in your code, you can open a read-only connection to the database as follows:

public static async void LoadDatabase()
{
    // Get a reference to the SQLite database
    ExternalStorageDevice _sdCard = 
        (await ExternalStorage.GetExternalStorageDevicesAsync())
        .FirstOrDefault();

    if (_sdCard == null)
    {
        MessageBox.Show("No SD Cards found");
        return;
    }
    else
    {
        ExternalStorageFile dbStoragefile = 
            await _sdCard.GetFileAsync("customers.sqlite");
        db = new SQLiteWinRT.Database(dbStoragefile.Path);
    }

    await db.OpenAsync(SQLiteWinRT.SqliteOpenMode.OpenRead);

    // PRAGMA temp_files=2 causes temporary files to be created in memory
    // rather than in a physicalfile in the same folder as the database 
    // Note that temp files only really get used if you are going to do 
    // complicated JOINs etc
    await db.ExecuteStatementAsync("PRAGMA temp_files=2");
}

Important: I had to modify the wrapper code posted up on http://sqlwinrt.codeplex.com in order to enable this usage. I added a new override of the Database object constructor that just takes the path to the database file as a string. Earlier versions had one override of the constructor that took a Windows.Storage.StorageFile parameter and another that took a Windows.Storage.StorageFolder and the filename as string, but neither of those were usable since the ExternalStorageFile object is not as you might expect a derivative of Windows.Storage.StorageFile, but is instead in the Microsoft.Phone.Storage namespace – hence I had to create a new constructor.

Download the latest version of the wrapper code from http://sqlwinrt.codeplex.com to get this update – note that this is built against SQLite version 3.8.2 – make sure you update to this version using the Visual Studio – Tools – Extensions and Updates wizard.

Note that it covers both scenarios covered in this post. At the top of app.xaml.cs there is a compile time symbol #define USE_SDCARD – comment this out to test the ‘database shipped as content’, or leave it in to test the SD card access. You’ll need a real device for the latter and before testing the app use your PC to copy the customers.sqlite file from the project onto the sd card and insert into your phone. Sadly, the emulator does not emulate a removable storage card!

And what a show! – great speakers, great food, large, comfortable venue, loads of sessions and an awesome party. Congratulations to NDC for putting on a great event, and I look forward to supporting them in future years.

My own contribution was two sessions: Windows Phone Networking Toolkit which was an update of a session I gave at TechEd this year, and which works through various topics related to effective network programming on Windows Phone, and Creating Killer Windows Phone Apps, a brand new session looking at ways to improve the quality of your apps. That went well and I look forward to repeating it somewhere soon. It looks at correct use of the animations in the Windows Phone Toolkit, programming Push Notifications using Windows Azure Notifications Hubs, localizing your app with the Multilingual Toolkit and reducing your memory footprint to improve your apps performance on low memory devices.

The slides for the sessions are available here:

I have been seeing a few comments by people complaining of poor performance when executing SELECT statements against a SQLite database using the SQLWinRT wrapper, or when adding new records using multiple consecutive INSERT statements.

With many relational databases, poor performance on INSERT can often be rectified just by wrapping all your operations within a single transaction. If you don’t explicitly declare a transaction when programming against SQLite, the database engine creates an implicit transaction for each statement execution and that has an overhead associated with it, so by wrapping all your INSERTs inside a single transaction, you would expect a performance boost.

With SELECT operations, poor performance is often just down to not having the correct keys defined. If you have a statement such as SELECT * FROM CUSTOMER WHERE NAME = ‘CONTOSO’ and you execute it on a database where you have not defined a key on the NAME column, the only way the database engine can find the required record is to start at the first record and read sequentially down the table until it reaches the end in order to select the rows to return (called a ‘Table Scan’). If you simply want to select all the records in a table (SELECT * FROM CUSTOMER), then you can’t avoid the Table Scan – in fact that’s what you want – so you would expect that there wasn’t very much you could do to improve performance in that case.

Well – that’s the theory. I set about building a simple demo to demonstrate these points, but it didn’t turn out quite how I expected! It turns out that, important though the points already mentioned are, the most important factor BY FAR is how you call the async methods of the SQLWinRT API! Read on…

The Test Program

For my test program, I was inspired by this post on Stack Overflow: How do I improve the performance of SQLite. That post is quite old and is talking about C++ dev, so not directly relevant to Windows Phone or Windows Store app development using managed code, but I liked the test case, which was to take a large dataset from the City of Toronto’s publically available transportation data. Unfortunately, the exact same dataset used in the original post isn’t available anymore, so I couldn’t do a direct comparison, but they did have the Transportation Trips dataset, containing 128982 records that we can first insert into a SQLite database, and then select them out again.

SQLite Bulk Insert

The dataset is in the form of a CSV text file, representing 128982 rows, each containing 8 columns. The code to read the file and extract the values  looks like this:

SQLiteWinRT.Database db;

private async void Button_Start_Click(object sender, RoutedEventArgs e)
{
    int Route_ID;
    int Service_ID;
    int Trip_ID;
    string Trip_Headsign;
    int Direction_ID;
    int Block_ID;
    int Shape_ID;
    bool Wheelchair_Accessible;

    Button_Start.IsEnabled = false;

    /*********************************************/
    /* Open the Database and create the Schema */
    await LoadDatabase();

    Stopwatch sw = Stopwatch.StartNew();
    int n = 0;

    // Prepare statement
    using (var statement = await db.PrepareStatementAsync(
        @"INSERT INTO TTC (id, Route_ID, Service_ID, Trip_ID, 
                           Trip_Headsign, Direction_ID, Block_ID, 
                           Shape_ID, Wheelchair_Accessible) 
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"))
    {
        // Start transaction
        await db.ExecuteStatementAsync("BEGIN TRANSACTION");

        /*********************************************/
        /* Open input file and import into Database*/
        var inputfile = await Windows.ApplicationModel.Package.Current.
            InstalledLocation.GetFileAsync("trips.txt");
        using (var inputStream = await inputfile.OpenSequentialReadAsync())
        {
            using (StreamReader rdr = 
                new StreamReader(inputStream.AsStreamForRead()))
            {
                while (true)
                {
                    string inputline = await rdr.ReadLineAsync();
                    if (inputline == null)
                        break;

                    // Discard line 0 - header line
                    if (n > 0)
                    {
                        string[] fields = inputline.Split(new char[] { ',' });

                        Route_ID = Int32.Parse(fields[0]); ;
                        Service_ID = Int32.Parse(fields[1]);
                        Trip_ID = Int32.Parse(fields[2]);
                        Trip_Headsign = fields[3];
                        Direction_ID = Int32.Parse(fields[4]);
                        Block_ID = Int32.Parse(fields[5]);
                        Shape_ID = Int32.Parse(fields[6]);
                        Wheelchair_Accessible = fields[7] == "1" ? true : false;

                        /* Bind parameter values and Insert */
                        statement.Reset();
                        statement.BindIntParameterAt(1, n);
                        statement.BindIntParameterAt(2, Route_ID);
                        statement.BindIntParameterAt(3, Service_ID);
                        statement.BindIntParameterAt(4, Trip_ID);
                        statement.BindTextParameterAt(5, Trip_Headsign);
                        statement.BindIntParameterAt(6, Direction_ID);
                        statement.BindIntParameterAt(7, Block_ID);
                        statement.BindIntParameterAt(8, Shape_ID);
                        statement.BindIntParameterAt(9, Wheelchair_Accessible ? 1 : 0);

                        await statement.StepAsync();
                    }

                    n++;

                    if (n % 100 == 0)
                    {
                        progressBar.Value = n;
                    }
                }
            }
        }

        // Commit transaction
        await db.ExecuteStatementAsync("COMMIT TRANSACTION");
    }

    db.Dispose();
    db = null;

    var report = String.Format(
        "Inserted {0:d} records in {1:0.00} seconds", 
        n - 1, 
        (double)sw.ElapsedMilliseconds / 1000);
    MessageBox.Show(report);

    Button_Start.IsEnabled = true;
}

private async Task LoadDatabase()
{
    db = new SQLiteWinRT.Database(
        ApplicationData.Current.LocalFolder, "sqliteperf.db");

    await db.OpenAsync();

    string sql = @"
        CREATE TABLE IF NOT EXISTS TTC 
            (id INTEGER PRIMARY KEY, 
                Route_ID INTEGER, 
                Service_ID INTEGER, 
                Trip_ID INTEGER, 
                Trip_Headsign TEXT,
                Direction_ID INTEGER, 
                Block_ID INTEGER, 
                Shape_ID INTEGER, 
                Wheelchair_Accessible BOOL)";

    await db.ExecuteStatementAsync(sql);
}

Using this code, and by commenting out lines here and there as appropriate, we can test three scenarios:

• Processing the input file but not inserting records in the database (this just gives us the time for the file processing)
• Uncomment the call to StepAsync() to Insert each record into the database individually
• Uncomment the lines executing the BEGIN TRANSACTION, END TRANSACTION statements so all records are inserted inside a single transaction

And the results? Well, not great, to be honest:

Although wrapping all the inserts into a single transaction halves the time it takes to insert the 128982 records compared to separate operations (where the database engine will activate an implicit transaction for every insert), we are still only getting 25ms per insert, or 40 operations per second. That’s not too impressive.

Trying ConfigureAwait(false)

Those performance figures are disappointing and make bulk insert operations in a Windows Phone or Windows Store app impractical. However, look what happens when we make a subtle change to the main loop of code:

Stopwatch sw = Stopwatch.StartNew();
int n = 0;

// Prepare statement
using (var statement = await db.PrepareStatementAsync(
    @"INSERT INTO TTC (id, Route_ID, Service_ID, Trip_ID, Trip_Headsign, 
                        Direction_ID, Block_ID, Shape_ID, Wheelchair_Accessible) 
                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"))
{
    // Start transaction
    await db.ExecuteStatementAsync("BEGIN TRANSACTION");

    /*********************************************/
    /* Open input file and import into Database*/
    var inputfile = await Windows.ApplicationModel.Package.Current.
        InstalledLocation.GetFileAsync("trips.txt");
    using (var inputStream = await inputfile.OpenSequentialReadAsync())
    {
        using (StreamReader rdr = new StreamReader(inputStream.AsStreamForRead()))
        {
            while (true)
            {
                string inputline = await rdr.ReadLineAsync();
                if (inputline == null)
                    break;

                // Discard line 0 - header line
                if (n > 0)
                {
                    string[] fields = inputline.Split(new char[] { ',' });

                    Route_ID = Int32.Parse(fields[0]); ;
                    Service_ID = Int32.Parse(fields[1]);
                    Trip_ID = Int32.Parse(fields[2]);
                    Trip_Headsign = fields[3];
                    Direction_ID = Int32.Parse(fields[4]);
                    Block_ID = Int32.Parse(fields[5]);
                    Shape_ID = Int32.Parse(fields[6]);
                    Wheelchair_Accessible = fields[7] == "1" ? true : false;

                    /* Bind parameter values and Insert */
                    statement.Reset();
                    statement.BindIntParameterAt(1, n);
                    statement.BindIntParameterAt(2, Route_ID);
                    statement.BindIntParameterAt(3, Service_ID);
                    statement.BindIntParameterAt(4, Trip_ID);
                    statement.BindTextParameterAt(5, Trip_Headsign);
                    statement.BindIntParameterAt(6, Direction_ID);
                    statement.BindIntParameterAt(7, Block_ID);
                    statement.BindIntParameterAt(8, Shape_ID);
                    statement.BindIntParameterAt(9, Wheelchair_Accessible ? 1 : 0);

                    await statement.StepAsync().AsTask().ConfigureAwait(false);
                }

                n++;
                if (n % 100 == 0)
                {
                    Dispatcher.BeginInvoke(()=>progressBar.Value = n);
                }
            }
        }
    }

    // Commit transaction
    await db.ExecuteStatementAsync("COMMIT TRANSACTION");
}

db.Dispose();
db = null;

Dispatcher.BeginInvoke(() =>
    {
        var report = String.Format(
            "Inserted {0:d} records in {1:0.00} seconds", 
            n - 1, (double)sw.ElapsedMilliseconds / 1000);
        MessageBox.Show(report);

        Button_Start.IsEnabled = true;
    });
}

The main – and significant change above – is that the StepAsync() method is now called in a different way:

await statement.StepAsync().AsTask().ConfigureAwait(false);

In addition, the code that ‘touches’ the UI, such as the MessageBox.Show() call, and the code setting the ProgressBar is now called using Dispatcher.BeginInvoke(Action a). This is necessary because the change to how StepAsync() is called has a knock-on effect of causing the code to end up executing on a background thread – I will explain more on this shortly.

What does this do to the timings?:

WHOA! What happened there? If you insert records within a transaction and use ConfigureAwait(false), you can achieve insert performance of one insert every 0.43 ms, or 2345 records per second. Now we’re talking! That means the performance of inserting records individually without using ConfigureAwait(false) is more than 144 times slower than the best case.

Why such a dramatic improvement? Well, the Task.ConfigureAwait(bool continueOnCapturedContext) method is used to configure the awaiter used to await the result of the asynchronous operation. An awaiter is an object you don’t see or program directly as a developer, but simply by using the await keyword when calling an asynchronous method, the compiler will generate code that uses an awaiter to handle the suspension and reactivation of your calling method while it waits for some asynchronous operation to complete. When the asynchronous method has completed, the awaiter resumes execution of your code which by default happens on the same context (think ‘thread’) you were on when you called the async method. But if you set ConfigureAwait(false), it doesn’t do that but instead continues on whatever context the async method executed on. There’s a performance advantage to this, which is why the advice to authors of library code is that you should always make async calls within your library code using ConfigureAwait(false) – the caller of your library method can then make the decision on whether to return to their original context by choosing whether or not to use ConfigureAwait(false) on the call to your async library method.

Which is what we are doing here – by setting ConfigureAwait(false) on our call to SqlWinRT:Statement.StepAsync() we are saying “don’t bother continue on the originating context, stay on whatever context you are executing on”. And as you can see, the performance gain is spectacular! To be honest, I was very surprised just how spectacular the performance improvement is. Normally, you would expect a modest performance gain, but I guess that because this call is happening between managed code and a WinRT component, the overhead of ConfigureAwait(true) - the default – is particularly significant.

One consequence of using ConfigureAwait(false) is that our code after the await call resumes on a background thread, so when we want to interact with the UI, such as to set the ProgressBar or to display the message at the end, we have to use the Dispatcher to make sure the code that touches the UI executes back on the UI thread:

Dispatcher.BeginInvoke(() =>
    {
        var report = String.Format("Inserted {0:d} records in {1:0.00} seconds", 
            n, (double)sw.ElapsedMilliseconds / 1000);
        MessageBox.Show(report);

        Button_Start.IsEnabled = true;
    });

If we don’t do that, you get an exception.

SELECT Performance

What about the performance of reading records out of the database? The warnings I gave in the introduction to this piece about having appropriate keys configured if you are doing a SELECT with a filter on a column that is not the primary key are still valid. But even if you are simply reading all the records out of a table, whether you use ConfigureAwait(false) again has a huge impact.

Here’s the code you would probably write to read all the rows in our table and to load them into in-memory objects in an ObservableCollection<T> ready to display on the UI:

private async void Button_Select_Click(object sender, RoutedEventArgs e)
{
    if (db == null)
    {
        db = new SQLiteWinRT.Database(
            ApplicationData.Current.LocalFolder, "sqliteperf.db");

        await db.OpenAsync();
    }


    Stopwatch sw = Stopwatch.StartNew();
    int n = 0;
    Button_Select.IsEnabled = false;

    string sql = @"
        SELECT
            id , 
                Route_ID , 
                Service_ID, 
                Trip_ID, 
                Trip_Headsign,
                Direction_ID, 
                Block_ID, 
                Shape_ID, 
                Wheelchair_Accessible
            FROM TTC";
    var stmt = await db.PrepareStatementAsync(sql);

    ObservableCollection<TransportationTrip> tripCollection =
        new ObservableCollection<TransportationTrip>();

    while (await stmt.StepAsync())
    {
        var trip = new TransportationTrip()
        {
            ID = stmt.GetIntAt(0),
            RouteID = stmt.GetIntAt(1),
            ServiceID = stmt.GetIntAt(2),
            TripID = stmt.GetIntAt(3),
            TripHeadsign = stmt.GetTextAt(4),
            DirectionID = stmt.GetIntAt(5),
            BlockID = stmt.GetIntAt(6),
            ShapeID = stmt.GetIntAt(7),
            WheelchairAccessible = 
                stmt.GetIntAt(8) == 1 ? true : false,
        };

        tripCollection.Add(trip);
        n++;
    }

    var report = String.Format("Selected {0:d} records in {1:0.00} seconds",
        n, (double)sw.ElapsedMilliseconds / 1000);
    MessageBox.Show(report);

    Button_Select.IsEnabled = true;
}

Running this gives the following result:

2838 seconds! That sucks – big time.

Then do it again, but make these subtle changes:

while (await stmt.StepAsync().AsTask().ConfigureAwait(false))
{
    var trip = new TransportationTrip()
    {
        ID = stmt.GetIntAt(0),
        RouteID = stmt.GetIntAt(1),
        ServiceID = stmt.GetIntAt(2),
        TripID = stmt.GetIntAt(3),
        TripHeadsign = stmt.GetTextAt(4),
        DirectionID = stmt.GetIntAt(5),
        BlockID = stmt.GetIntAt(6),
        ShapeID = stmt.GetIntAt(7),
        WheelchairAccessible = stmt.GetIntAt(8) == 1 ? true:false,
    };

    tripCollection.Add(trip);
    n++;
}

Dispatcher.BeginInvoke(() =>
    {
        var report = String.Format("Selected {0:d} records in {1:0.00} seconds", 
            n, (double)sw.ElapsedMilliseconds / 1000);
        MessageBox.Show(report);

        Button_Select.IsEnabled = true;
    });

Which gives this result:

That’s some improvement! Comparing the two:

That’s a huge improvement!

[WARNING: The timings quoted here are for running on the emulator. You should expect times on a real device to be very substantially slower.]

Conclusions

Some golden rules emerge from this:

1. Use ConfigureAwait(false) when calling SqlWinRT async methods, particularly within repetitive looping such as INSERT or SELECT of a significant number of records. Remember that when you do so, your code will not resume on the same context it started on, so take appropriate measures to handle this.
2. In addition to using ConfigureAwait(false), wrap large numbers of INSERTs in a single transaction to get the best performance.

The last learning from this is not specific to SqlWinRT: if you call async methods on a WinRT component, there is a significant overhead in switching back to the originating context. Try calling such a component using ConfigureAwait(false) - it may yield significant performance benefits.

Lastly – this was a test program. If you ever find yourself trying to read 128000 records into memory, I would suggest you have another think about your app design and whether you really need all those objects in memory .

The WPF desktop app allows the ‘operator’ to select a city, temperature and weather conditions and then generate tile, toast and raw notification messages to communicate the weather forecast to the phone. The WPF app does rather more than that and in addition implements two other key parts of a push notifications scenario, as it also hosts a WCF service and maintains a list of subscribers, simulating the cloud service that you would build for a real Push Notifications solution.

In summary, the phone registers for push notifications¹ and receives a unique URI (the channel URI) which it then sends off to your cloud service² (the WCF service) and the cloud service maintains a list of all subscribers for this app. Then the backend logic (the WPF UI in this demo) reads the list of subscriber channel URIs when it has something to communicate to the phone and sends the tile/toast/raw payloads to the subscriber URIs³. The subscriber URIs are actually endpoints hosted on MPNS (Microsoft Push Notification Services), so MPNS actually takes care of delivering the payloads to the phone⁴.

Fine – except that sample is difficult to setup and run, since you have to run Visual Studio as administrator otherwise the WCF self-hosting doesn’t work, you have to open port 8000 in your firewall because that is the port the WCF service listens on, and you have to edit the phone app code to set the correct hostname of the PC where the WCF service is running. Those are all implementation difficulties, but the real crime of this demo is that it’s too simplistic a demo and gives the impression that writing a Push Notifications solution is easier than it really is. In a real world app, when you post to MPNS you get back a response that gives you information about the state of the phone app subscription. There’s a whole page in the documentation about what you might get back: Push Notification Service response codes for Windows Phone and even if you ignore most of that, at the very least you should be removing subscribers from your list if you get a 404 Not Found response back, indicating that the subscription channel URI you posted to is no longer valid.

In fact, writing backend logic to send push notifications and correctly react to the Push Notification Service response codes is hard. And if you want to send push notifications to more than one client platform, such as Windows 8, iOS and Android, it’s incredibly hard.

That is – unless you use Windows Azure Notification Hubs

Windows Azure Notification Hubs

Windows Azure Notification Hubs is a service running in Windows Azure that makes it really easy to send push notifications to mobile clients. You can read an overview here, but in essence what it does is remove the need to manage the subscriber list and create your own subscription WCF service from your backend logic and provide an easy to program API for sending notifications – and not only to Windows Phone, but to Windows 8, iOS and Android clients too.

But the benefits of Notification Hubs are not only for those building a cross-platform solution. Even if you are building a push notifications app for a single type of client, using this technology will greatly ease the creation of your solution.

I thought it would be interesting to do a new version of the Weather sample, but using Windows Azure Notification Hubs to make a comparison between the two ways of doing it, so for the rest of this post, I will walk you through what I did (download the sample code from the link at the bottom of this post).

From an architectural point of view, the main thing is that the custom WCF service and subscriber list disappears completely. Great, no need to build and host a custom web service in order to build a push solution! Instead, the notification hub fulfils that purpose and acts as a gateway between your backend logic and MPNS.

Configuring the Notification Hub

First stage in building this is to configure the notification hub in Windows Azure, which is all done using the portal. You need to have a Windows Azure account and then go to the portal at http://manage.windowsazure.com.

1. Click +NEW at the bottom of the screen.
2. Click App Services, then Service Bus, then Notification Hub, then Quick Create.
3. Type a name for your notification hub – I chose weathernotificationhub, select your desired region – West Europe for me, then click Create a new Notification Hub.
   
   
4. You will now see the Service Bus namespace created screen.
   
   
   Now, before you leave this screen, click on the Install the Windows Azure SDK (includes Service Bus client libraries) link to make sure you’ve got the client libraries installed in Visual Studio that you will need in both the phone app and also the desktop app.
   
5. Click Connection Information at the bottom. Take note of the two connection strings – you will need them in a moment.
   

That’s it. Your Windows Azure Notification Hub is configured and ready to go.

Connecting the Phone App to Windows Azure Notification Hubs

Now let’s turn to the phone app. There aren’t many changes we need to make from the phone app we used in pre-notification hubs days. It still needs to register for push notifications using Microsoft.Phone.Notifications.HttpNotificationChannel and register for toast, tile and/or raw notifications. And when the app opens the channel, it will get a new Channel Uri reported to it when the ChannelUriUpdated event fires.

In the original app, it was at this point that you sent the channel Uri off to the WCF service hosted in the Weather Service desktop app, but instead we now send that off to our Windows Azure Notification Hub. To do this, you need to add a reference to the client libraries using the WindowsAzure.Messaging.Managed NuGet package.

In the Visual Studio menu, click Tools, then Library Package Manager, then Package Manager Console. Then, in the console window type:

Install-Package WindowsAzure.Messaging.Managed

and press Enter.

Now in the phone app, in MainPage.xaml.cs, we add the using Microsoft.WindowsAzure.Messaging using statement at the top, and then modify the ChennelUriUpdated event handler as follows:

async void httpChannel_ChannelUriUpdated(object sender, NotificationChannelUriEventArgs e)
{
    Trace("Channel opened. Got Uri:\n" + httpChannel.ChannelUri.ToString());
    Dispatcher.BeginInvoke(() => SaveChannelInfo());

    Trace("Subscribing to channel events");
    await SubscribeToServiceAsync();
    SubscribeToNotifications();

    Dispatcher.BeginInvoke(() => UpdateStatus("Channel created successfully"));
}

The SubscribeToServiceAsync() method is a new version of SubscribeToService() in the original demo. Instead of calling the WCF service (the commented out code), it now instantiates a Microsoft.WindowsAzure.Messaging.NotificationHub instance, which takes two arguments: the name of my notification hub (weathernotificationhub) and the connection string called DefaultListenSharedAccessSignature that you get from the Connection Strings display in the portal (step 5 in the previous section).

private async Task SubscribeToServiceAsync()
{
    try
    {
        var hub = new NotificationHub(
            "weathernotificationhub",
            "Endpoint=sb://weathernotificationhub-ns.servicebus.windows.net...[your value] ...");

        Registration registration = 
            await hub.RegisterNativeAsync(httpChannel.ChannelUri.ToString());
        Trace("Registered with Azure Notification Hub, Registration Id:" 
            + registration.RegistrationId);

        Dispatcher.BeginInvoke(() => 
            UpdateStatus("Registered with Windows Azure Notification Hub"));
    }
    catch (RegistrationAuthorizationException rEx)
    {
        Trace(rEx.Message);
        throw;
    }
}

That’s it as far as the phone client goes. Everything else stays the same: push notifications are delivered by the notifications infrastructure and handled in exactly the same way. Windows Azure Notification Hubs simply replaces your own subscription service, as far as the phone client goes.

Sending Notifications From Your Backend Logic

This is where the true benefit of Windows Azure Notification Hubs shines through. You no longer have to create your own web service with table storage to receive the channel URIs from your phone client subscribers, nor do you have to write logic to examine the status returns you get back from MPNS when you post notification messages and prune out stale registrations from the subscriber lists. You don’t need to write logic to post to each subscriber channel URI individually, just one call to the notification hub will cause the message to be sent out to all your clients. And if your backend logic needs to support Windows 8, iOS or Android clients as well as Windows Phone, the benefits of  Windows Azure Notification Hubs multiplies a hundred-fold.

Getting Registration Details from the Hub

In the Weather app, I started by deleting everything to do with the WCF registration service that we used in the previous version. Then fixed up the UI where it used to display things like “Waiting for connection…”. One thing the original did, was show the current count of registrations to the registration service. To do the same with Notification Hubs, you can call the NotificationHubClient.GetAllRegistrationsAsync(int32 top) method which returns details of all registrations limited to the top ‘n’ by the value you pass in the argument. In order to use the NotificationHubClient object, you need to add a reference to the client libraries using the WindowsAzure.Messaging.Managed NuGet package just the same as we did with the phone app.

After adding the NuGet package to my sample app, we show the number of clients on the screen using the following method:

private async Task UpdateRegistrationsCountAsync()
{
    NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString(
        "Endpoint=sb://weathernotificationhub-ns.servicebus.windows.net/;…",
        "weathernotificationhub");

    var registrationsCount = await hub.GetAllRegistrationsAsync(Int32.MaxValue);
    await Dispatcher.BeginInvoke((Action)(() => 
      { txtRegistrationsCount.Text = registrationsCount.Count().ToString(); }) 
    );
}

The NotificationHubClient constructor takes two arguments: the first is the connection string you get from the Notification Hub Connection Strings page in the Windows Azure portal, but unlike the phone client which used the DefaultListenSharedAccessSignature, this time you need the DefaultFullSharedAccessSignature as this app is doing full interaction with the notification hub, such as posting notifications. The second arg is the name of your notification hub.

Sending Toast and Tile Push Notifications

Now to the main purpose of the app. Most of the logic around formatting and sending notification messages is encapsulated into the NotificationSenderUtility class in my sample. For example, this contains the following method to format the XML to send for a Windows Phone Toast notification:

private static string prepareToastPayload(string text1, string text2)
{
    // Create encoding manually in order to prevent
    // creation of leading BOM (Byte Order Mark) xFEFF at start
    // of string created from the XML
    Encoding Utf8 = new UTF8Encoding(false); // Prevents creation of BOM
    MemoryStream stream = new MemoryStream();
    XmlWriterSettings settings = new XmlWriterSettings() 
                { 
                    Indent = false,
                //   Encoding = Encoding.UTF8    !!NO-> adds Unicode BOM to start
                    Encoding = Utf8,    // Use manually created UTF8 encoding
                };
    XmlWriter writer = XmlWriter.Create(stream, settings);
    writer.WriteStartDocument();
    writer.WriteStartElement("wp", "Notification", "WPNotification");
    writer.WriteStartElement("wp", "Toast", "WPNotification");
    writer.WriteStartElement("wp", "Text1", "WPNotification");
    writer.WriteValue(text1);
    writer.WriteEndElement();
    writer.WriteStartElement("wp", "Text2", "WPNotification");
    writer.WriteValue(text2);
    writer.WriteEndElement();
    writer.WriteEndElement();
    writer.WriteEndDocument();
    writer.Close();

    return  Encoding.UTF8.GetString(stream.ToArray());
}

WARNING: While building this, I discovered that Windows Azure Notification Hubs are quite a bit fussier about the XML you send it than MPNS actually is. The original version of the code used the standard Encoding.UTF8 in the XmlWriterSettings (commented out in the code above) which results in an invisible Byte Order Mark (BOM) character being inserted at the front of the string, which causes Windows Azure Notification Hubs to barf when you send it that. The code above creates the XML without the BOM at the front which works fine. To read more about this, see my blog post BOMbin’ the L (aka Wierd Errors with XmlWriter).

Having prepared the payload, to send a Toast or Tile is simply a case of calling the NotificationHubClient.SendMpnsNativeNotificationAsync(string payload) method. The implementation in the attached sample is slightly more complex than this, but essentially it’s the following:

    NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString(
        "Endpoint=sb://weathernotificationhub-ns.servicebus.windows.net/;...",
        "weathernotificationhub");

    NotificationOutcome outcome = await hub.SendMpnsNativeNotificationAsync(payload);

Sending Raw Notifications

The other thing the original sample does is send raw notifications directly to the app (if it happens to be running of course). The body of a raw payload is entirely up to the app developer, since unlike toast and tile notifications that may be processed by the Push Notifications handler in the phone OS, raw notifications are only ever handled by the app. So in the new version of this sample, the code building the raw notification payload is almost the same as before:

private static string prepareRAWPayload(string location, string temperature, string weatherType)
{
    // Create encoding manually in order to prevent
    // creation of leading BOM (Byte Order Mark) xFEFF at start
    // of string created from the XML
    Encoding Utf8 = new UTF8Encoding(false); // Prevents creation of BOM
    MemoryStream stream = new MemoryStream();
    XmlWriterSettings settings = new XmlWriterSettings()
    {
        Indent = false,
        //   Encoding = Encoding.UTF8    !!NO-> adds Unicode BOM to start
        Encoding = Utf8,    // Use manually created UTF8 encoding
    };
    XmlWriter writer = XmlTextWriter.Create(stream, settings);

    writer.WriteStartDocument();
    writer.WriteStartElement("WeatherUpdate");

    writer.WriteStartElement("Location");
    writer.WriteValue(location);
    writer.WriteEndElement();

    writer.WriteStartElement("Temperature");
    writer.WriteValue(temperature);
    writer.WriteEndElement();

    writer.WriteStartElement("WeatherType");
    writer.WriteValue(weatherType);
    writer.WriteEndElement();

    writer.WriteStartElement("LastUpdated");
    writer.WriteValue(DateTime.Now.ToString());
    writer.WriteEndElement();

    writer.WriteEndElement();
    writer.WriteEndDocument();
    writer.Close();

    return Encoding.UTF8.GetString(stream.ToArray());
}

I then had to do a bit of digging to figure out how to send this raw notification via Windows Azure Notification Hubs. One of the many methods of NotificationHubClient is SendNotificationAsync(Notification notification) which is a more generic Send* method than SendMpnsNativeNotificationAsync that we used before. It has the advantage that we can add HTTP headers to the request we send to MPNS, so we can define the X-NOTIFICATION header that a raw notification requires:

        // For raw notifications, have to use the more generic SendNotificationAsync() method
        Notification notification = new MpnsNotification(payload);
        notification.Headers.Add("X-NotificationClass", "3"); // Required header for RAW
        outcome = await hub.SendNotificationAsync(notification);

Summary

Hopefully this has given you an introduction to building a push notification solution using Windows Azure Notification Hubs. There is plenty more to discover as I have only scratched the surface. For example, this sample sends notifications to all subscribers, but you can send to only a subset of subscribers using Tags. And of course, you can also send to Windows 8, iOS and Android clients, formatting the payloads for each client family yourself and then calling the SendWNSNativeNotificationAsync, SendAPSNativeNotificationAsync or SendGCMNativeNotificationAsync methods. An even easier way to do this, is to use the Templates feature, which is a neat way of sending a single generic message from your backend logic and allowing Windows Azure Notification Hubs to format the native message payload for each platform for you.

Here are some useful resources to allow you to dig deeper into Windows Azure Notification Hubs:

• Windows Azure Notification Hubs Overview
• Get started with Notification Hubs – Windows Phone
• How To: Windows Azure Notification Hubs (Windows Store Apps)
• How To: Windows Azure Notification Hubs (iOS Apps)
• How To: Windows Azure Notification Hubs (Android Apps)
• Use Notification Hubs to send breaking news (shows how to use tags to send push notifications to different groups of subscribers)

You should also be aware that Windows Azure Mobile Services uses Notification Hubs to send push notifications.

You can download the sample code described here yourself using the link at the bottom of this post. You will have to create your own notification hub and plug in your own hub name and connection strings, both in MainPage.xaml.cs in the phone project, and in MainWindows.xaml.cs in the desktop app – you should be able to find where easily enough.

IMPORTANT: In Visual Studio, go to Tools – Options – Package Manager and check Allow NuGet to download missing packages during build. After you open the solution, right-click on the project and then click Manage NuGet Packages. When the package manager window opens, you will see a bar across the top saying that some NuGet packages are missing – click the Restore button to download the required NuGet packages.

Download sample code: Windows Azure Notifications Hubs Weather Demo

The dev tools have always been free (download the Windows Phone 8.0 SDK for free from http://dev.windowsphone.com), but if you wanted to test your app on the emulator, you had to have a PC with a Core i3/i5/i7 processor and have Windows 8 Pro or Enterprise, which are the editions that include the necessary Hyper-V support. So if you couldn’t meet those hardware and OS specs, your only option was to run your app on a real phone – but to use a real phone you had to have registered for a developer account at http://dev.windowsphone.com so you could unlock it, which is normally $99/year (although that is currently on special offer at $19 until August 26th). At this point, the ‘mildly curious developer’, faced with the prospect of having to part with money to try out their new app, can lose interest and of course we’ve lost a potential convert to our developer community.

Any dev can unlock one phone for testing without a registration fee

Now, anyone can unlock one phone to test their apps without needing a dev account, and can install up to two ‘work in progress’ apps onto the phone at any one time. Fantastic! So now you can get started with Windows Phone dev using *any* PC running Windows 8 (any edition) at no charge (well, you need a handset obviously ). Only caveat is that your PC and Windows 8 OS still needs to be 64 bit, as that is a requirement of the SDK.

However, we’re seeing a lot of postings in the forums saying that their app only works intermittently, or works fine in debug but then fails intermittently when deployed. Quite often, the reason for this is that the developer has inadvertently called some async methods in a ‘fire and forget’ fashion, and what they are seeing is a synchronization problem; they assume that some async code has completed, and some code elsewhere is trying to use the result value from the async method before the async method has completed. The result, a null reference exception or incorrect behaviour because some object is not fully populated.

The problem with async/await in Application_Launching and Application_Activated

I accidently encountered this problem when I was building a simple demo for the last Windows Phone 8 JumpStart video series. OK, it was a dumb demo, but demos are often dumb because you want to keep them simple so as not to distract from the main point you are trying to make. This demo was to show how to program with the new Windows.Storage apis, so in my App class Application_Launching event handler I had code that read some data out of a file and saved the string into a public property called LogData:

    public partial class App : Application
    {
        ...

        // This is shared amongst all the pages
        // It is the contents of the log itself, as a large string
        public string LogData;

        // Code to execute when the application is launching (eg, from Start)
        // This code will not execute when the application is reactivated
        private async void Application_Launching(object sender, LaunchingEventArgs e)
        {            
            // Warning - this async method may not complete before this event handler 
            // method completes and other events fire. 
            this.LogData = await LoadFromLocalFolderAsync();
        }

        public async static Task<string> LoadFromLocalFolderAsync()
        {
            string theData = string.Empty;

            // There's no FileExists method in WinRT, so have to try to get a reference to it
            // and catch the exception instead
            StorageFile storageFile = null;
            bool fileExists = false;
            try
            {
                // See if file exists
                storageFile = await Windows.Storage.StorageFile.GetFileFromApplicationUriAsync(
                    new Uri("ms-appdata:///local/CaptainsLog.store"));
                fileExists = true;
            }
            catch (FileNotFoundException)
            {
                // File doesn't exist
                fileExists = false;
            }

            if (!fileExists)
            {
                // Initialise the return data
                theData = string.Empty;
            }
            else
            {
                // File does exist, so open it and read the contents
                Stream readStream = await storageFile.OpenStreamForReadAsync();
                using (StreamReader reader = new StreamReader(readStream))
                {
                    theData = await reader.ReadToEndAsync();
                }
            }

            return theData;
        }

And in MainPage.xaml.cs, OnNavigatedTo, the code gets the value of LogData from the App class and sets the Text property of a TextBlock to the string to show the contents of the file on the screen. Logic elsewhere in the app writes content into the file, so on startup you should always see the current contents of the file shown on the screen on application launch.

        protected override void  OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
        {
             base.OnNavigatedTo(e);
            
            // When we navigate to the page - put the log text into the display
            completeLogTextBlock.Text = ((App)App.Current).LogData;
        }

What could possibly go wrong? Well, plenty as it turns out! The first time you run this app, the screen is empty which is to be expected because the file is empty. But then you use the app to enter some text which is written to the file, then close the app. Launch again, and where you would expect to see some text displayed, the screen is still empty! What gives?

Exploring execution of Async methods

To show why this went wrong, let’s look at a simple console app to explore the two ways you can execute an async method.

using System;
using System.Threading.Tasks;

namespace AsyncSynchronizationProblem
{
    class Program
    {
        static void Main(string[] args)
        {
            new TheClassThatDoesWork().DoTheWork();

            Console.ReadKey();
        }
    }

    class TheClassThatDoesWork
    {
        public async void DoTheWork()
        {
            Console.WriteLine("TRACE Calling SomeLongProcedure Async");
            await SomeLongProcedureAsync();
            Console.WriteLine("TRACE Returned from SomeLongProcedure Async");

            Console.WriteLine();

            Console.WriteLine("TRACE Calling SomeLongProcedure Fire and forget");
            SomeLongProcedureAsync();
            Console.WriteLine("TRACE Returned SomeLongProcedure Fire and forget");
        }

        private Task SomeLongProcedureAsync()
        {
            var task = Task.Run(() =>
                {
                    Console.WriteLine("TRACE Starting SomeLongProcedure work");
                    System.Threading.Thread.Sleep(1000); // Simulate long running op
                    Console.WriteLine("TRACE Completed SomeLongProcedure work");
                });

            return task;
        }
    }
}

When you run this app, you see the following console output:

In this sample, the long running method, Task SomeLongProcedureAsync() simply sleeps for 1s to simulate executing some lengthy code. In the method DoTheWork(), we just call the async method twice, once using await and once without. We write to the console output to show what is happening. I’m sure it is no surprise to most of you, that when we call using await, the caller waits until the long-running work has completed before continuing execution itself:

TRACE Calling SomeLongProcedure Async
TRACE Starting SomeLongProcedure work
TRACE Completed SomeLongProcedure work
TRACE Returned from SomeLongProcedure Async

and when we call without using await, it works in a fire-and-forget fashion, and the caller continues execution without waiting for the task to complete:

TRACE Calling SomeLongProcedure Fire and forget
TRACE Starting SomeLongProcedure work
TRACE Returned from SomeLongProcedure Fire and forget
TRACE Completed SomeLongProcedure work

This is pretty basic stuff about calling async methods. So why is this relevant?

Dangers of making event handlers async

Let’s look at another example. This one is still a console app, but has four classes, Program which just hosts the Main method of the console app, App, PhoneApplicationService and MainPage – I have named the last three the same as key classes in a Windows Phone app for reasons which hopefully will become obvious.

using System;
using System.Threading.Tasks;

namespace AsyncSynchronizationProblemWithEvents
{
    class Program
    {
        static void Main(string[] args)
        {
            var phoneAppService = new PhoneApplicationService();
            var app = new App(phoneAppService);
            
            phoneAppService.DoTheWork();

            // When the PhoneApplicationService has completed, construct a MainPage instance
            var mainPage = new MainPage();
            // Call methods on it
            mainPage.OnNavigatedTo();

            Console.ReadKey();
        }
    }

    class App
    {
        private PhoneApplicationService appService;

        public App(PhoneApplicationService applicationService)
        {
            appService = applicationService;
            // Subscribe to the PhoneApplicationService lifetime event(s)
            appService.ApplicationLaunching += Application_Launching;
        }

        async void Application_Launching(object sender, EventArgs e)
        {
            Console.WriteLine(
                "TRACE Event handler - Calling SomeLongProcedure Async");
            await SomeLongProcedureAsync();
            Console.WriteLine(
                "TRACE Event handler - Returned from SomeLongProcedure Async");
        }

        private Task SomeLongProcedureAsync()
        {
            var task = Task.Run(() =>
            {
                Console.WriteLine("TRACE Starting SomeLongProcedure work");
                System.Threading.Thread.Sleep(1000); // Simulate long running op
                Console.WriteLine("TRACE Completed SomeLongProcedure work");
            });

            return task;
        }
    }

    class PhoneApplicationService
    {
        public event EventHandler ApplicationLaunching;

        public void DoTheWork()
        {
            // Here we do some stuff...

            // Call any subscribers to our event(s)
            if (ApplicationLaunching != null)
            {
                ApplicationLaunching(this, new EventArgs());
            }
        }
    }

    class MainPage
    {
        public void OnNavigatedTo()
        {
            Console.WriteLine("TRACE Starting MainPage OnNavigatedTo");

            // Do some work here too...

            Console.WriteLine("TRACE Completed MainPage OnNavigatedTo");
        }
    }
}

If you study this app, you will see that the Main method creates instances of PhoneApplicationService, creates an instance of App and passes the PhoneApplicationService instance into its constructor, and finally creates an instance of MainPage and calls its OnNavigatedTo method. This is, of course, exactly what happens when you start a Windows Phone app. When you run it, the output looks like this:

There is clearly something odd going on here: the trace messages from the execution of SomeLongProcedure are output, but right in the middle, the calls to MainPage OnNavigatedTo appear. What’s going on?

Inside the App constructor, we hook the ApplicationLaunching event of the PhoneApplicationService instance so that in the event handler we can do some setup work when the app launches, and this of course is where we call the long-running async method:

        async void Application_Launching(object sender, EventArgs e)
        {
            Console.WriteLine(
                "TRACE Event handler - Calling SomeLongProcedure Async");
            await SomeLongProcedureAsync();
            Console.WriteLine(
                "TRACE Event handler - Returned from SomeLongProcedure Async");
        }

We are using the await keyword inside this event handler, so of course we mark it with the async modifier as normal. So at first glance, the developer expects this code to execute and for the event handler code to wait for completion of SomeLongProcedureAsync() before continuing execution, which in this case is simply to write a message to the console. If we look at the output, that is indeed what happens:

TRACE Event handler - Calling SomeLongProcedure Async
TRACE Starting SomeLongProcedure work
TRACE Starting MainPage OnNavigatedTo
TRACE Completed MainPage OnNavigatedTo
TRACE Completed SomeLongProcedure work
TRACE Event handler - Returned from SomeLongProcedure Async

But as you can see from the trace output, the MainPage OnNavigatedTo method is executed before the Application_Launching event handler has completed, even though it was clearly invoked by the Main function after the PhoneApplicationService.DoWork() method was invoked, which is the method that fires the Application_Launching event:

        static void Main(string[] args)
        {
            var phoneAppService = new PhoneApplicationService();
            var app = new App(phoneAppService);
            
            phoneAppService.DoTheWork();

            // When the PhoneApplicationService has completed, construct a MainPage
            var mainPage = new MainPage();
            // Call methods on it
            mainPage.OnNavigatedTo();

            Console.ReadKey();
        }

The problem of course, is that there is no way of awaiting the completion of an event handler, so when the DoWork() method fires the Application_Launching event, it does so in a fire and forget fashion, just the same as in the calling of SomeLongProcedureAsync() without using the await keyword, as we demonstrated in the first example. During execution, code executes sequentially until we hit the await keyword inside the Application_Launching event handler, at which point the wonders of the Task Parallel Library ensure that execution of the Task continues without blocking the original context and control returns immediately to the caller (PhoneApplicationService.DoTheWork() in this case). 

If the Application_Launching event handler had returned async Task instead of async void: 

async Task Application_Launching(object sender, EventArgs e)

(which is not possible with event handlers) and if it had been called using the await keyword, then  DoWork() would have awaited completion of the Application_Launching event handler logic before continuing, in which case, MainPage.OnNavigatedTo() would not have been called before the Application_Launching logic had completed. You can only await completion of methods that return Task or Task<T> and since this is impossible with event handlers, you can never await their completion. Hence there is every chance that your MainPage OnNavigatedTo logic will execute before your Application_Launching or Application_Activated logic has completed, although it is actually hit or miss, which is why so many developers report intermittent failures.

The Solution?

Of course, this is not a new problem. Any time you execute code asynchronously from your application lifecycle event handlers, whether using async/await or other asynchronous programming techniques, there is a real chance that you will have to implement some kind of synchronisation logic to ensure that your page logic does not try to access objects before your startup logic has set them up. Using older async coding technologies, this was perhaps a little more obvious to the developer, but many devs use the new Task-based async methods without thinking too deeply about it. It just works – or at least seems to until you hit a timing-related problem such as the one I’ve described here.

In the simple demo I described at the beginning of this article, the solution was simple. Instead of exposing the data fro the app in a public property of the App class, I created a new ViewModel class to host it and used XAML databinding to bind the TextBlock on MainPage to a property of the ViewModel. When the Application_Launching logic completed, it updated the ViewModel instance, and through the wonders of INotifyPropertyChanged, the MainPage UI updated with the correct data.

For other solutions, you may need to implement some other mechanism. One example was shown me by Peter Torr of the Windows Phone Product Group and is shown in the sample app which you can download from my post on SQLite:  SQLite-WinRT: Database programming on Windows Phone and Windows 8 . That sample opens (and the first time, creates) the SQLite database in Application_Launching and when resuming from tombstoning in Application_Activated. That sample used an old-fashioned synchronisation primitive, a ManualResetEvent, as a flag that is only set when the application lifecycle logic has finished:

  public partial class App : Application
  {
    static SQLiteWinRTPhone.Database db;
    static ManualResetEvent DBLoaded = new ManualResetEvent(false);

    public static Task<SQLiteWinRTPhone.Database> GetDatabaseAsync()
    {
      return Task.Run(() =>
      {
        DBLoaded.WaitOne();
        return db;
      });
    }

    // Code to execute when the application is launching (eg, from Start)
    // This code will not execute when the application is reactivated
    private void Application_Launching(object sender, LaunchingEventArgs e)
    {
      LoadDatabase();
    }

    private async Task LoadDatabase()
    {
      // Get a reference to the SQLite database
      db = new SQLiteWinRTPhone.Database(
          ApplicationData.Current.LocalFolder, "sqlitedemo.db");

      await db.OpenAsync();

      string sql = @"CREATE TABLE IF NOT EXISTS
                                Customer (Id      INTEGER PRIMARY KEY,
                                          Name    VARCHAR( 140 ),
                                          City    VARCHAR( 140 ),
                                          Contact VARCHAR( 140 ) 
                            );";
      string description = "Create Customer table";
      await db.ExecuteStatementAsync(sql);
        

      DBLoaded.Set();
    }
}

The ManualResetEvent is declared unsignalled as a static field of the app class:

 static ManualResetEvent DBLoaded = new ManualResetEvent(false);

and is only set at the very end of the async Task LoadDatabase() method:

private async Task LoadDatabase()
{
  // Get a reference to the SQLite database
  db = new SQLiteWinRTPhone.Database(…);
  await db.OpenAsync();
  ...

  DBLoaded.Set();
}

The final key ingredient is a ‘gatekeeper’ method which must be called by every other method in the app that wants to access the database:

public static Task<SQLiteWinRTPhone.Database> GetDatabaseAsync()
{
  return Task.Run(() =>
  {
    DBLoaded.WaitOne();
    return db;
  });
}

This method waits for the DBLoaded semaphore to be set and does so on a Threadpool thread (through Task.Run()) so does not block the UI thread.

Calling async code from Application_Deactivated or Application_Closing

The guidance here is “don’t'”. If you write your apps carefully, you can be saving changes to persistent data as you go along, so you shouldn’t have anything to do in the application lifecycle events.

If you must, you can try doing something like this:

SomeAsyncMethod().AsTask().Wait()   

If the operation completes within the timeout period AND it doesn’t deadlock due to needing to pump the UI thread, it will work… but don’t count on it.

The session is a deep dive on the new SQLite-WinRT API that we released on Codeplex last week. This new API is a thin wrapper around the native database engine, sqlite3.dll and exposes the key methods of the database engine through a modern Windows Runtime API callable from C#, Visual Basic and C++ that is all async methods and which reports unexpected results by throwing exceptions. The code is all up at http://sqlwinrt.codeplex.com. There’s no download for this project – you just download the source code, which is a complete Visual Studio 2012 solution containing projects for Windows Phone 8 and Windows 8 Store App components.

Why another database API on Windows Phone?

That’s a valid question! After all, we already have the Local Database in the Windows Phone SDK which is a LINQ to SQL API where the underlying database is SQL Server Compact Edition 3.7-ish. And we also have the SQLite-NET API for accessing SQLite which also exposes a LINQ API for working with a SQLite database. The Local Database LINQ to SQL API is not available for Windows 8 Store App development so not a great choice if you want to share code between your phone and tablet/PC apps, but SQLite-NET is available for both phone and Windows 8, as is SQLite-WinRT – so why do we need this new API?

SQLite-WinRT is for those developers who prefer to work with SQL statements rather than the object-oriented layer that LINQ gives you. It is also for those coming to Windows Phone 8 or Windows 8 from other platforms who have worked with SQLite before probably programming with C/C++ – the methods exposed by the SQLite-WinRT API are very thin wrappers around the SQLite C/C++ API. It’s all about giving choice to our developers – we have heard feedback that not all developers like to work with a local database using LINQ, so this API is for them.

To give you a comparison, this is how you might select a Customer from the Customer table using SQLite-NET:

var db =  new SQLite.SQLiteAsyncConnection(App.DBPath); 
var _customer = await  (from c in db.Table<Customer>() 
                        where c.Id == customerId  
                        select c).FirstOrDefaultAsync(); 

if (customer != null) 
{  
    var Id = _customer.Id;
    var Name = _customer.Name;
}

And this is how you would do the same thing in SQLite-WinRT:

using (var db = new SQLiteWinRTPhone.Database(
    ApplicationData.Current.LocalFolder, "demo.db")) 
{ 
    await db.OpenAsync(); 
    using (var stmt = await db.PrepareStatementAsync("SELECT name, age FROM people")) 
    { 
        while (await stmt.StepAsync()) 
        { 
            var name = stmt.GetTextAt(0); 
            var age = stmt.GetIntAt(1); 
        } 
    } 
}

The choice is yours!

Getting Started with SQLite-WinRT

Start by installing the SQLite Extension SDK. This is the native dll, sqlite3.dll, which contains the database engine. To install this, in Visual Studio, on the Tools menu, open Extensions and Updates. In the left hand pane, select Online – Visual Studio Gallery, then enter SQLite into the search box. Select SQLite for Windows Phone and click the Download button to download and install the vsix package. (The same procedure works for the SQLite for Windows Runtime extension for Windows 8 Store Apps.)

After you’ve installed that, the native sqlite3 dlls are installed into a folder under C:\Program Files (x86)\Microsoft SDKs\Windows Phone\v8.0\ExtensionSDKs\SQLite.WP80\. That’s just for your information – you should never have to manually copy the sqlite3.dll from there in order to use the database

Warning: You will be prompted whenever an update is available for the SQLite database engine. When you install it, you will inadvertently ‘break’ SQLite-WinRT, in the sense that Visual Studio will be unable to open the project. That is because the project references the C:\Program Files (x86)\Microsoft SDKs\Windows Phone\v8.0\ExtensionSDKs\SQLite.WP80\3.7.17 path in a number of places, and when you upgrade SQLite, the version part of that path will change.
I will be adding a powershell script into the download to make it easy to update the SQLite-WinRT project file with the correct path, but if this happens to you before I do it, just edit the SQLiteWinRTPhone.vcxproj file (carefully!) to correct references to the old version and update them to pojnt at whatever new version you have installed.

Creating your Project

Create your C# or Visual Basic project as normal. Then go to http://sqlwinrt.codeplex.com and download the source code from there – remember there is no ‘official download’ for this Codeplex project.

The source code for this project is a solution containing two projects, the SQLite-WinRT component for Windows Phone, and the same thing but for Windows 8 Store Apps. Copy the Windows Phone project and use ‘Add Existing Project…’ to add it into your solution, then add a reference to the SQLite-WinRT project from your C#/VB project. That’s all you have to do!

Remember, if Visual Studio cannot load the SQLite-WinRT project, it’s probably because the SQLite extension has been updated. Read the Warning a few paragraphs before to resolve this.

Creating a database

To create a database, you can use code such as this:

    private async void LoadDatabase()
    {
      // Get a reference to the SQLite database
      db = new SQLiteWinRTPhone.Database(ApplicationData.Current.LocalFolder, "sqlitedemo.db");

      await db.OpenAsync();

      string sql = @"CREATE TABLE IF NOT EXISTS
                                Customer (Id      INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
                                          Name    VARCHAR( 140 ),
                                          City    VARCHAR( 140 ),
                                          Contact VARCHAR( 140 ) 
                            );";
      string description = "Create Customer table";
      await ExecuteSQLStatement(db, sql, description);

      sql = @"CREATE TABLE IF NOT EXISTS
                   Project (Id          INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
                            CustomerId  INTEGER,
                            Name        VARCHAR( 140 ),
                            Description VARCHAR( 140 ),
                            DueDate     DATETIME,
                            FOREIGN KEY(CustomerId) REFERENCES Customer(Id) ON DELETE CASCADE 
                            )";
      description = "Create Project table";
      await ExecuteSQLStatement(db, sql, description);

      // Turn on Foreign Key constraints
      sql = @"PRAGMA foreign_keys = ON";
      description = "Enable foreign key constraints";
      await ExecuteSQLStatement(db, sql, description);

      DBLoaded.Set();
    }

    private static async Task ExecuteSQLStatement(
                                SQLiteWinRTPhone.Database db, string sql, string description)
    {
      try
      {
        await db.ExecuteStatementAsync(sql);
        Debug.WriteLine(description + " executed OK");
      }
      catch (Exception ex)
      {
        var result = SQLiteWinRTPhone.Database.GetSqliteErrorCode(ex.HResult);
        throw new ApplicationException(description + " Failed with error " + result);
      }
    }

You can call this code from your Application_Loaded and Application_Activated event handlers. One particular thing I want to call out here: notice the last line of the LoadDatabase method:

DBLoaded.Set();

DBLoaded is a ManualResetEvent which is used to flag when the database opening logic has completed. It is declared at the top of my App.Xaml.cs class, along with a ‘Gatekeeper’ method called GetDatabaseAsync():

  public partial class App : Application
  {
    static SQLiteWinRTPhone.Database db;
    static ManualResetEvent DBLoaded = new ManualResetEvent(false);

    public static Task<SQLiteWinRTPhone.Database> GetDatabaseAsync()
    {
      return Task.Run(() =>
      {
        DBLoaded.WaitOne();
        return db;
      });
    }

The ManualResetEvent is created unset, and doesn’t get set until the last line of LoadDatabase(). All other database access code in my viewmodel classes makes a call to GetDatabaseAsync as the first thing it does. The DBLoaded.WaitOne() call in there blocks until DBLoaded is set, so this prevents any other database accessing code from executing until the LoadDatabase() method has completed.

Handling Errors

The SQLWinRT API throws exceptions when something goes wrong. Unfortunately, exceptions thrown in a WinRT component are not the most developer-friendly things ever devised. They are COMException objects, have no useful error text in them and just return to you a HResult. But you can get a meaningful error message if you call the handy GetSqliteErrorCode method which is used like this:

      try
      {
        await db.ExecuteStatementAsync(sql);
      }
      catch (COMException ex)
      {
        var result = SQLiteWinRTPhone.Database.GetSqliteErrorCode(ex.HResult);
        throw new ApplicationException(description + " Failed with error " + result);
      }

GetSqliteErrorCode() returns the – yes, you guessed it – SQLite error code, which mostly are self explanatory (such as ‘SQLITE_CONSTRAINT’) are documented at http://www.sqlite.org/c3ref/c_abort.html.

CRUD operations

Create

To create records in the database, use code similar to this:

try 
{
    // Connection already opened in app.xaml.cs - get reference
    SQLiteWinRT.Database db = await App.GetDatabaseAsync();
    using (var custstmt = await db.PrepareStatementAsync(
      "INSERT INTO Customer (Name, City, Contact) VALUES (@name, @city, @contact)"))
    {  
        // NOTE that named parameters have a leading "@",":" or "$".         
        custstmt.BindTextParameterWithName("@name", customer.Name);
        custstmt.BindTextParameterWithName("@city", customer.City);
        custstmt.BindTextParameterWithName("@contact", customer.Contact);

        // Use StepAsync to execute a prepared statement 
        await custstmt.StepAsync();
     } 
} 
catch (System.Runtime.InteropServices.COMException) 
{ … }

This uses a SQL statement containing named parameters (@name, @city, @contact), which you prepare using PrepareStatementAsync() and which you ‘plug in’ the values using the BindtypeParameterWithName(string parametername, type value) method. An alternative is to use anonymous parameters (shown below when describing Update).

Notice that we use the StepAsync() method to execute a row-returning SQL statement. StepAsync may seem curious, but the SQLite C/C++ API method this wraps is called Step(), so that’s the name we adopt for our WinRT method.

IMPORTANT: Remember to dispose of your Statement objects when you’re done with them – a good way of doing that is to use the C# using statement, as shown above.

Read

To read objects, use a SELECT statement:

public async Task<CustomerViewModel> GetCustomerAsync(int customerId)
{
    var db = await App.GetDatabaseAsync();
    CustomerViewModel customer = null;
    using (var readstmt = await db.PrepareStatementAsync(
              "SELECT Id, Name, City, Contact FROM Customer WHERE Id = " + customerId)) 
    {
      if (await readstmt.StepAsync() == true)
        {
       var customer = new CustomerViewModel()
              {
      Id = readstmt.GetIntAt(0),
                 Name = readstmt.GetTextAt(1),
            City = readstmt.GetTextAt(2),
          Contact = readstmt.GetTextAt(3)
        }; 
        }
    }
    return customers;
}

StepAsync() returns true if there is a row to return. The example above would reasonably be expected to return just a single row, but if you get multiple rows returned from the database, then you would loop around creating your in-memory objects (your viewmodel objects) and entering them into some collection until StepAsync() returns false.

An alternative form of SELECT allows you to extract data from the rowset returned from the database by using the column name, rather than the column position as the previous code sample did. You enable this behaviour by calling the Statement.EnableColumnsProperty() method. Then the results are returned as a Dictionary<string, string> which you key into using the column name:

    using (var readstmt = await db.PrepareStatementAsync(
              "SELECT Id, Name, City, Contact FROM Customer WHERE Id = " + customerId))
    {
         // Enable the columns property
         statement.EnableColumnsProperty();

       if (await readstmt.StepAsync() == true)
        { 
            // it isn't super useful as all columns are returned as text and must be parsed
            var columns = statement.Columns;
            var customer = new CustomerViewModel()
              {
      Id = int.Parse(columns["Id"]),
                 Name = columns["Name"],
            City = columns["City"],
          Contact = columns["Contact"]
        }; 
        }
    } 

This does impose a slight performance overhead, and every value returns as a string, so you have to parse the string to get the correct value type, as shown here for the Id column.

Update

Update is simple enough. This sample uses anonymous parameters in the SQL statement, rather then the named parameters we saw when discussing the Insert operation. Note that anonymous parameters are 1-based, not 0-based like just about every other collection in .NET!

// See if the customer already exists
var existingCustomer = await GetCustomerAsync(customer.Id); 
if (existingCustomer != null) 
{    using (var custstmt = await db.PrepareStatementAsync(
        "UPDATE Customer SET Name = ?, City = ?, Contact = ? WHERE Id=?")) 
    { 
        // NOTE when using anonymous parameters the first has an index of 1, not 0. 
        custstmt.BindTextParameterAt(1, customer.Name);
   custstmt.BindTextParameterAt(2, customer.City);
    custstmt.BindTextParameterAt(3, customer.Contact);
    custstmt.BindIntParameterAt(4, customer.Id);
        await custstmt.StepAsync();
   }
}

Delete

Delete is simple enough:

  string sql = @"DELETE FROM Customer WHERE Id={0}"; 
  sql = string.Format(sql, customerId);

  // Can use ExecuteStatementAsync to run non row returning statements
  await App.db.ExecuteStatementAsync(sql);

Summary

That’s an introduction to getting started with SQLite-WinRT. In the next posts, I will talk some more about Foreign Key constraints and about working with Read-only databases.

• Original
• Innovative
• Easy to use
• Engaging and visually appealing to the user

Full details on the Windows Phone Dev Blog here, entry page for the Windows Phone Porting Challenge competition here.