If you have any questions about any of the content on the blog, feel free to send me a message on Twitter on the handle @paulwhit, or respond in the post’s comments.

Thanks for reading!

I make mobile apps professionally (as a Microsoft consultant, I’ve worked on Windows and mobile web apps) and as a hobby. My primary hobby app, which I maintain as a service for the military enthusiast, survivalist and Army communities, is Army Field Manuals. It provides a library of a bunch of PDF files published by the U.S. Army.

I’ve also recently completed my MIM degree, in which had some close encounters with librarian- and archivist-type people. These are dedicated folks who organize and classify information, and many of them are extremely tech-savvy. This grad school work inspired me to want to do a better job than the Army has done to expose the public contents of Army doctrine. The primary ways I’ve come up with to do this are to integrate a variety of sources, provide some manual classification, and provide a robust search interface.

How do I do full-text search in a mobile (Windows) application?

Search in my app currently only searches the titles (and soon I’ll be adding descriptions). Unlocking the full text of the documents for search requires (at least in Microsoft lingo) an IFilter for PDFs. The technology is built-in with Windows 8.1, which provides hooks through IndexableContent,, but since the 3GB of field manuals are only downloaded to the app clients on demand, there’s no way to provide full text searching across the whole corpus.

Enter SharePoint Search. SharePoint can search PDFs natively, can provide full text results with hit highlighting, and can return the metadata associated with the document in context. It can also provide rank tuning (in my case, I might want to do things like rank highly rated documents (LibraryThing has some ratings)). So, blending all these ideas I had an epiphany that SharePoint could provide some good value for my app community. So I’ve been figuring out how this might work and be affordable.

What about SharePoint Online for Office 365?

I’ve tried this, but the anonymous site you get seems to preclude using the search API from an anonymous user. When I run this query logged in, it works. As anonymous I get a file not found error. In the SharePoint Foundation solution below it works fine, so I’m assuming this is a specific exclusion of functionality in the public site on Office 365.

/_api/search/query?querytext=%27army%27&QueryTemplatePropertiesUrl=%27spfile://webroot/queryparametertemplate.xml%27&SummaryLength=360&KeywordInclusion=%271%27

What’s the cheapest way to use SharePoint’s API layer?

Shared hosting of SharePoint Foundation is cheapest. Even though no CALs are required for External Users in the other versions, you’d need a CAL for yourself at least—and where would you purchase that as a hobbyist developer? And after that, the 3rd party hosting providers charge a lot more money for standard or enterprise licensed SKUs. I’m still discovering the features that SharePoint Foundation can do, since I typically work with the Enterprise SKU of SharePoint. However, I’ve signed up for a 3GB plan (enough to hold all the public Army doctrine PDF documents) on PlexHosted.com (affiliate link) for about $25/mo, and I’ve found that I can add site columns, access the search REST API anonymously, and access the lists API as well. So far so good. The Term Store is not available; I was hoping to use that to help tag content. However, a multiselect lookup will work for what I’m doing in the short term.

What are the limitations?

There’s more to APIs than anonymously exposing metadata and search results. This solution doesn’t do authentication or much of anything in the way of security or protections for the system against abuse. There is some logging, however, so hopefully that could prove useful. Also the system will be on a shared instance to keep the costs manageable. Not sure if that would scale. Another down side that I’ve found is that my search API access is lumped in with some other sites’ anonymous results. That doesn’t have any real impact for me, but it’s definitely not secured across site boundaries.

In order to make this happen, you have to go through a few hoops. Here is a sample event handler that wires up the pieces together including converting to a PNG image to bring the file size down. See MSDN’s “How to share an image” article for configuration.

In this case, we have a ScrollViewer in an ListView control (it’s used in a continuous document viewing section of an app). The method shares what is currently visible on the screen.

private async void OnDataRequested(DataTransferManager sender, DataRequestedEventArgs e)
{
    var scrollViewer = RecurseChildren<ScrollViewer>(lvImages).FirstOrDefault();

    if (scrollViewer != null)
     {
         DataRequest request = e.Request;
         request.Data.Properties.Title = "Clip from an app";

        DataRequestDeferral deferral = request.GetDeferral();

        try
         {
             var bmp = new RenderTargetBitmap();
             await bmp.RenderAsync(scrollViewer);

            var pixelBuffer = await bmp.GetPixelsAsync();

            var reader = DataReader.FromBuffer(pixelBuffer);
             var bytes = new byte[pixelBuffer.Length];
             reader.ReadBytes(bytes);

            var randomAccessStream = new InMemoryRandomAccessStream();
             var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, randomAccessStream);
             encoder.SetPixelData(BitmapPixelFormat.Rgba8, BitmapAlphaMode.Straight, (uint)bmp.PixelWidth, (uint)bmp.PixelHeight, 96, 96, bytes);
             await encoder.FlushAsync();
             await randomAccessStream.FlushAsync();
                         
             request.Data.SetBitmap(RandomAccessStreamReference.CreateFromStream(randomAccessStream));
         }
         finally
         {
             deferral.Complete();
         }
     }
 }

Here is an example from my Army Field Manuals app. Note that the share item doesn’t include the app border, which is handy in this case:

Additional references used:

• “Windows 8.1 RenderTargetBitmap” by Geoff Webber-Cross
• “How to get byte[] from IBuffer” answered by Min Zhu
• XAML render to bitmap sample (Windows 8.1) on MSDN

Note: This blog post is based on Windows 8.1 RTM and Visual Studio 2013 RC.

Microsoft released a componentized, high-performing PDF viewing component for XAML/C# applications on 4/2/2014. Woohoo! This serves as a good reference sample of how this needs to be done (C# front-end with a C++ back-end).

If you want more bells and whistles, a number of third parties add more to the reader experience such as alternate viewing methods (facing page vs. continuous view, etc). Figure 1 shows a breakdown of all the components that I have seen. These are all expensive (from $495 to “if you have to ask, it’s too expensive”), so they might not work well for a hobbyist app developer. But if you have a need in your commercial or LOB application, you may find one of these to be useful. 

Fig 1. PDF Viewing Components for Windows Store applications

* Offers two rendering modes – one that converts to XAML and another that uses the PDF API
** Windows 8.1 Update offers the ability for enterprise apps to broker desktop components for use in WinRT applications.

If you know of others, please post a reply or send me a tweet @paulwhit. I’m actively monitoring this topic.

Also Ahmed-Faoud has ported MuPDF for use in Windows Store applications. This is very fast, but carries a GPL license.

If high performance viewing of large or complex documents isn’t a consideration for your use case, you could provide rasterization to bitmap images either within your application at runtime or on a load event or background task, or by using a batch process like below. You could then load them in a ListView control and you should get pretty good performance. Rasterizing the PDF pages on the fly in a ListView is probably going to be too slow, especially on a low-powered device.

As an example, I’m using some C# code in a batch application to generate thumbnails for my Army Field Manuals app. I’m distributing the images with my app, so there’s no processor use on the client.

To rasterize the first page of a PDF document as a thumbnail, in a Windows Store application, add this to a button click event handler. The “file” variable is a file path. A quick way to access a local file is to put it in your %localappdata%\packages\{app id}\LocalState path and use ApplicationData.Current.LocalFolder to retrieve its file name.

var pdfDoc = await PdfDocument.LoadFromFileAsync(file);
var pdfPage = pdfDoc.GetPage(0);

var pageImageFile = await ApplicationData.Current.LocalFolder.CreateFileAsync(
"sm_" + Path.GetFileNameWithoutExtension(file.Path) + ".scale-100.png", CreationCollisionOption.ReplaceExisting);
var randomStream = await pageImageFile.OpenAsync(FileAccessMode.ReadWrite);
PdfPageRenderOptions options = new PdfPageRenderOptions();
options.DestinationWidth = 250;
await pdfPage.RenderToStreamAsync(randomStream, options);
await randomStream.FlushAsync();

randomStream.Dispose();
pdfPage.Dispose();

There is a team dedicated to the PDF APIs at Microsoft. You can find the PDF API team’s blog led by program manager Shalu Gupta on TechNet.

For the record, I am currently using DevExpress in my Army Field Manuals app.

private async Task UnzipWithProgress(StorageFolder outFolder, Stream zipFileStream, ProgressBar pb)
{
    bool bDelete = false;

    using (var archive = new ZipArchive(zipFileStream))
    {
        try
        {
            // Ensure the UI update part runs in the UI thread
            var ignore = this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
            {
                this.tbDownloadStatus.Text = "Unzipping";
                pb.Value = 0;
                pb.Maximum = archive.Entries.Count;
            });

            foreach (ZipArchiveEntry entry in archive.Entries)
            {
                if (entry.Name != "")
                {
                    string fileName = entry.FullName.Replace("/", @"\");
                    StorageFile newFile = await CreateFile(outFolder, fileName);
                    Stream newFileStream = await newFile.OpenStreamForWriteAsync();
                    Stream fileData = entry.Open();
                    byte[] data = new byte[entry.Length];
                    int len = await fileData.ReadAsync(data, 0, data.Length);
                    while (len > 0)
                    {
                        await newFileStream.WriteAsync(data, 0, len);
                        len = await fileData.ReadAsync(data, 0, data.Length);
                    }
                    await newFileStream.FlushAsync();

                    var ignore2 = this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
                    {
                        pb.Value++;
                    });
                }
            }

            var itemId = OriginalUriString;
            this.Frame.Navigate(typeof(ItemDetailPage), itemId);
        }
        catch (Exception ex)
        {
            bDelete = true;
        }
        finally
        {
            if (bDelete)
                outFolder.DeleteAsync();
        }
    }
}

To use it, call it this way. In my case I called it from the LoadState method.

StorageFile file;
try
{
    file = await ApplicationData.Current.LocalFolder.GetFileAsync(OriginalUriString);
}
catch (FileNotFoundException)
{
    file = null;
}

if (file != null)
{
    using (var ws = await file.OpenReadAsync())
    using (var forReadRandomStream = ws.AsStreamForRead(0))
    {
        string path = ApplicationData.Current.LocalFolder.Path + @"\" + Path.GetFileNameWithoutExtension(file.Name) + @"\";
        await UnzipWithProgress(await StorageFolder.GetFolderFromPathAsync(path), forReadRandomStream, progressBar1);
    }
}

Apologize that this is not a File->New demo, but I thought it might be useful and Scott Hanselman made me do it.

Looks like this may be the future of Windows Live Writer (according to the tea leaves). It feels a lot like Writer initially. It can do a lot out of the box but I'll spare you the needless includes from Flickr and other locations. Although I don't see Insert->Code, maybe cut and paste from Visual Studio is formatted well enough.

Even though I'm a Microsoft employee, I haven't been privy to anything with this release—in fact, I don't think I was able to glean anything other than what's been posted online. I'm looking forward to learning more in the next few weeks and months. I'm particularly intrigued by the sync story with Skydrive and the investment in web versions of the apps. Also fascinating to think about is 3^rd party apps integrated with Office 365, Skydrive, and Office Web Apps. Looks like there's an Office store where folks can acquire these add-ons.

Here are a few questions that I have after an hour working with some of the release. There seems to be a cohesive connected-device picture coming together.

Word 2013 Questions

• Will the sharing features (such as blog posting) be available in the web app?
• Will there be an RSS reader online to compete with Google Reader?
• Will real-time collaboration be backward compatible with Office 2010?
• How well does the PDF editor work?

Outlook 2013 Questions

• I imported my Google Reader RSS feeds OPML file. Now, when I click on RSS Feeds, it goes to my Google Reader. Is this a fluke?
• Anything new in people merging? Will my merged contacts sync with the Metro Mail App and Windows Phone? It's a lot of work to redo these associations.
• I see you can add Facebook and LinkedIn social connectors. Where's Twitter?
• Are live tiles coming?

OneNote 2013 Questions

• Can OneNote edit PDFs like Word can?

Other/Random

• Why aren't we seeing the Metro style versions in this beta release?
• Any new authentication mechanisms planned for Skydrive (Facebook?)

I'll probably figure some of this out, but if anyone has some of this information already, feel free to include it in the comments.

Thanks!

p.s. Man this thing posts a LOT faster than Writer.

I’d been looking into developing an app for Windows Phone since I picked up the Samsung Focus 18 months ago – to learn the technology and showcase what the platform can do. As a Microsoft employee, I feel tied to the platform, but also felt like it could be approachable since I am familiar with the language and some of the tools already. I also liked what I had heard about the Windows Phone 7.5 update and wanted to try out some of the features. I recently finished a work project using WPF in which I picked up some XAML experience, so I felt energized about wrapping my head around the Silverlight-based development stack, and becoming a published app author.

Behind the Idea

So I set out to brainstorm ideas. I figured that good mobile app either has an interesting data set or an interesting algorithm, and is something that is useful away from your computer. I thought about places where you might go and need an app, and somehow remembered about a trip that my dad and I took to the Vietnam Veterans Memorial Wall in Washington, DC. While at the Wall, my dad tried to remember some of the folks he knew about who died in the war. We thumbed through some arcane books where the names were listed alphabetically and by hometown, and found one of the fallen GIs. But we had trouble finding others. Given that most of the soldiers, sailors, airmen, marines and coast guardsmen perished or were declared M.I.A. over 40 years ago, it seems like recall could be a problem worth trying to solve.

The Tiny Wall

So grew the idea of having a portable version of the Vietnam Wall database. I contacted multiple organizations that maintain the database of names behind the wall (there are a few parallel copies maintained by different organizations). There was a lot of interest. Even though the number of potential users is relatively small compared to iPhone or Android, the consensus was that the additional interest and exposure for the wall, and the memory of those listed on it, would justify the efforts. I chose to partner with the good folks at the Virtual Wall®, who tirelessly maintain a virtual memorial for each war casualty and M.I.A., and provide updates to their copy of the data. With their agreement to provide a regularly-updated dump of the wall names (surprisingly, even after 40 years, the data is updated monthly with dozens of changes), I agreed to develop the app free of charge and not include ads or solicit donations. The focus would be on providing a new way to search the database, and providing additional information about the wall to visitors including a link to The Virtual Wall®’s memorials.

We decided on an initial set of features:

• Support searching the wall by First and Last Name
• Support searching by Home of Record (City/State/Country)
• Enable “sounds like” searching for names
• When searching by Home of Record, include nearby locations
• Indicate where on the wall the name is located

I chose some technical requirements in order to make the platform shine:

• Use the Windows Phone embedded database included with Windows Phone 7.5
• Adhere to Metro design language principles as best as possible, and take design cues from others (related, make it look good)
• Learn and use the standard tools: Visual Studio, Windows Phone Tools, Expression Blend, various CodePlex projects
• Avoid common mistakes: slow scrolling list boxes, inconsistent fonts, ignoring user color choices
  

The Published App

After about 200 hours of work, Vietnam Wall was published just after Memorial Day 2012. I was shooting for a release for that holiday, but it didn’t quite get approved in time.

http://www.windowsphone.com/en-US/apps/0b758fa2-6414-4381-923e-557172cb3edc?wa=wsignin1.0

Tools and Resources

Here is a collection of tools and resources I used in the process of creating the Vietnam Wall app. In retrospect, I’d say the tools are robust and capable, if you know where to look. In order to create a clean and compelling app, it’s necessary to be conscious about all your pixels. Vietnam Wall isn’t a perfect design by any means, but I think it looks clean and works well, thanks mostly to Scott Hanselman and Jeff Wilcox, as well as the other links I’ve shared below.

Essential Reading

These resources are key to getting started and knowing about the tools and processes to develop Windows Phone apps. Scott Hanselman’s volume is quite helpful, although after the 10th or so 6-hour session working on Vietnam Wall, I begin to curse his blog post’s title. Ha!

• From Concept to Code in 6 hours: Shipping my first Windows Phone App by Scott Hanselman
• Charles Petzold’s free e-Book, Programming Windows Phone 7
• App Hub community forum’s “look here to find the resources that you need” post
• 80% of Windows Phone Developers Not Using the Free Tools
  

Database

I used a Mango Local database. Turns out it’s not quite SQL CE 3.5, and you have to create and populate it in a very specific way, by creating a Windows Phone app and using the emulator to pump the data in using LINQ-to-SQL, then copying it from local storage to your “real” app. It seems like a rather convoluted process, but it works OK. In the end, it’s nice to have full LINQ access to a data store and not have to distribute any libraries (since the SQLCE drivers are included in the Windows Phone 7.5 OS).

• Windows Phone Mango Local Database (SQL CE) by WindowsPhoneGeek (awesome series)
• LINQ-to-SQL CompiledQuery
• MSDN: Query Examples (LINQ to SQL)
• How to Migrate/Downsize a SQL Server database to SQL Server Compact 4.0 (and 3.5)
  

Interface Controls

I used with Telerik RadControls. As a Microsoft employee, I was able to snag a free (normally $99) license. I was happy with their support and the tile, progress bar and animation helper controls (as part of their custom Root Application Frame) were slick. In particular, I like how transition animations work when paging away from the tile controls (see it in action in Vietnam Wall). I also used their Diagnostics and Picker controls, which I think saved some effort over the CodePlex versions.

• Omni-present progress bar (template for Silverlight)
• WP7 LongListSelector in depth | Part2: Data binding scenarios
• Telerik Windows Phone 7 Controls Forums
• Telerik RadControls for Windows Phone documentation
  

Design

I figure I spent 60% of my time getting this app to look decent. Jeff Wilcox’s design guides for developers were infinitely helpful here, and I set out to apply his guidance as best I could. I also thought the MSDN resources were decent.

• Jeff Wilcox’s “Metro” design guide for developers, v1.00 by Jeff Wilcox
• A quick look at MetroRadio- quick spacing, margin, and icon tips for Windows Phone devs by Jeff Wilcox
• MetroGridHelper- A helpful debugging assistant for designers and developers alike by Jeff Wilcox
• Windows Phone – Important Design Guidelines
• Using WP7 Accent Color Effectively
• MSDN: Essential Graphics, Visual Indicators and Notifications for Windows Phone
• MSDN: User Experience Design Guidelines for Windows Phone
• CESPage.com Windows Phone 7 Icons
  

Conclusion

I had a pretty good experience developing my first app for Windows Phone. I bit off a fairly complex project (using a Local DB and providing some advanced search functionality), and ended up spending considerable time learning and applying the Metro design language and various Windows Phone design guidelines. I’d say the tools are fairly easy and good to work with (especially the Visual Studio/Blend pass-off), and the documentation is abundant and easy to find. Now that Vietnam Wall is out the door, developing additional apps will have a lot smaller learning curve.

On the surface, one might think this would be frustrating. Feeling micromanaged, like your time revolves around the numbers. If the numbers became out of sync with business reality, or if they were unreasonable, I might feel this way. But Microsoft leadership seems to be responsive to changes and suggestions each year, and the process gets me regularly thinking about creating goals that help grow our business, increase my impact, and help others succeed (among other things).

I work in a super competitive place. My compensation is partially awarded in the context of others in my pay grade, as well as the one above me. This means some very experienced, passionate, and intelligent people are vying for the same recognition that I am.

So each year I sit down a few times with my manager to tweak my “SMART” goals to help me stand out above my peers. In some ways, it’s a personal exercise rather than an organizational one. For instance, trying to figure out how to put a personal metric on increasing my exposure and influence is a little tough, and the output seems a little goofy. Statements about number of technical discussions outside my immediate team, for instance, represent my intention rather than a metric tied directly to business accomplishments. I think the point is more to consistently think about accomplishing the intention of goals, rather than the language itself.

The process isn’t flawless, and some of the metrics that get assigned to me may not fit exactly with my work conditions. Also, the goals are not explicitly examined as if they were an exam, so some aggressive goals I set (such as a new MCITP or MCPD certification) get lost in the shuffle if I don’t highlight them specifically. But, generally speaking, goal setting at MCS is a motivational process and has helped lead me down an interesting path towards a more visible presence within my team and at my customers.

My justification for attending was mostly technical: to fill some gaps and learn new skills, although I also expected to make new contacts and friends. I’ve done these things, but I also gained some unexpected insight. Namely, I was inspired to focus and document my career plans.

This “aha” moment began with a “Birds of a Feather” lunch group. This type of lunch group at PASS involves choosing a lunch table assigned a topic. My choice was “Why you need to distinguish yourself from the rest,” a group facilitated by DBA and professional development guru Edwin Sarmiento (@bassplayerdoc). We discussed the motivations to stand out, and the topic quickly shifted to figuring out how to stand out. We had great discussions about how to change perceptions that others have of you, and how to do some personal branding in order to be recognized for your work.

I attended some great technical presentations, and although all the presenters have impressive backgrounds, two presenters stood out to me as role models for interesting careers and deep focus: Stacia Misner (@staciamisner) and Joy Mundy. They both have an intense passion and deep focus on the Business Intelligence stack and many years of experience.

Finally, Andy Warren (@sqlandy) presented on developing a Professional Development Plan. In the talk, he outlined the need to discover and document the resources you plan to use to better your career, and described how to create a written plan. This was excellent and is something I have started working on. Importantly, he made points about balance and avoiding burn-out, and I plan to use this to develop insights into my own development plan.

My head is swimming in new thoughts and ideas, and I’m working to develop a story for my next 5-10 years. I have some role models to keep tabs on and some tools to help me move forward. Thanks PASS!

As a consultant for Microsoft, I am required to become an expert on technologies as they emerge. Since I work with customers that are sometimes slow to adopt technologies, I need to ramp up on older technologies as well. Continuously learning multiple generations of technology can be a fun challenge, although sometimes daunting!

Microsoft provides me with amazing tools to stay fresh. MCS allows for 4 weeks of paid training per year, and every 18 months, I attend TechReady, a week-long training in Seattle. I also have free access to everything Microsoft Learning and Microsoft Press creates. I try to take full advantage of these tools and always add certification exams to my yearly commitments in order to keep myself on track.

With all the need for me to learn technologies, there seems to be little time to cycle back to foundational knowledge that can help me make broader decisions, and help customers make more impactful technology choices that are led by business needs.

In the next 5 years, my impact will increase. I’ve made my technology skills and aptitude for running with new technology challenges evident on every project I’ve been assigned to, and I plan to continue to do so.

With the knowledge I gain from MIM program as well as my growing impact on the job, I am hoping to position myself to more fully affect change at a higher level.

In the next 5 years, I hope to shift from the “tactical” impact of designing and implementing specific systems to the world of “strategic” thinking, and I think this degree program will help me get to that point.