• Source code: https://github.com/ljw1004/blog/tree/master/VSIX/TextAdornmentVsix

1. Control Panel > Add Remove Programs > Microsoft Visual Studio 2015 > Modify
2. Common Tools > Visual Studio Extensibility Tools > Ok

1. Visual Studio > Tools > Extensions and Updates > Online
2. Search for ".NET Compiler Platform SDK" > install it, and restart VS

1. File > New > Project > Visual C# > Extensibility > VSIX Project
2. This has created some files which we no longer need: delete index.html and stylesheet.css from your project

1. Project > References > Add Reference > Framework > check System, PresentationCore, PresentationFramework, WindowsBase > OK
2. Tools > Options > NuGet Package Manager > General > Automatically check for missing packages during build > ensure this button is checked > OK
3. Right-click on project > Add > New Item > Visual C# Items > Web > JSON file, and call it "project.json"
4. Edit the project json and replace its contents with what's below
5. Right-click on project > Unload Project
6. Right-click on project > Reload Project
7. Build > Build Solution. This will produce warnings "Detected package downgrade System.Collections.Immutable from 1.2.0 to 1.1.37" -- these warnings are fine, and indeed necessary (see below). But if there are errors about .NETFramework version, that's because of a mismatch between net46 specified in the project.json "frameworks" section and the .NET version in your Project > Properties > Application > TargetFramework. The minimum target is .net46. You should make sure both are in sync.
8. Open source.extension.vsixmanifest > Assets > New, type = "MefComponent", source = "A project in current solution", and specify the project.
9. The NuGet version numbers are specific to an update to VS2015: the versions "1.3.x" and "14.3.x" listed here require at least Update3; you should switch to the older "1.2.x" and "14.2.x" for something that will run on Update2.

{
    "dependencies": {
        "Microsoft.CodeAnalysis": "1.3.0",
        "Microsoft.VisualStudio.LanguageServices": "1.3.0",
        "Microsoft.VisualStudio.Shell.14.0": "14.3.25407",
        "Microsoft.VisualStudio.Text.UI.Wpf": "14.3.25407",
        "System.Collections.Immutable": "1.1.37",
        "System.ComponentModel": "4.0.1",
        "VSSDK.ComponentModelHost": "12.0.4",
        "VSSDK.DTE": "7.0.4"
    },
    "frameworks": {
        "net46": { }
    },
    "runtimes": {
        "win": { }
    }
}

1. Right-click on your project > Add > Class
2. Replace its contents with the following.

using System.Linq;

namespace MefRegistration
{
    using System.ComponentModel.Composition;
    using Microsoft.VisualStudio.Utilities;
    using Microsoft.VisualStudio.Text.Editor;

    [Export(typeof(IWpfTextViewCreationListener)), ContentType("text"), TextViewRole(PredefinedTextViewRoles.Document)]
    public sealed class TextAdornment1TextViewCreationListener : IWpfTextViewCreationListener
    {
        // This class will be instantiated the first time a text document is opened. (That's because our VSIX manifest
        // lists this project, i.e. this DLL, and VS scans all such listed DLLs to find all types with the right attributes).
        // The TextViewCreated event will be raised each time a text document tab is created. It won't be
        // raised for subsequent re-activation of an existing document tab.
        public void TextViewCreated(IWpfTextView textView) => new TextAdornment1(textView);

#pragma warning disable CS0169 // C# warning "the field editorAdornmentLayer is never used" -- but it is used, by MEF!
        [Export(typeof(AdornmentLayerDefinition)), Name("TextAdornment1"), Order(After = PredefinedAdornmentLayers.Selection, Before = Microsoft.VisualStudio.Text.Editor.PredefinedAdornmentLayers.Text)]
        private AdornmentLayerDefinition editorAdornmentLayer;
#pragma warning restore CS0169
    }
}


public sealed class TextAdornment1
{
    private readonly Microsoft.VisualStudio.Text.Editor.IWpfTextView View;
    private Microsoft.CodeAnalysis.Workspace Workspace;
    private Microsoft.CodeAnalysis.DocumentId DocumentId;
    private System.Windows.Controls.TextBlock Adornment;

    public TextAdornment1(Microsoft.VisualStudio.Text.Editor.IWpfTextView view)
    {
        var componentModel = (Microsoft.VisualStudio.ComponentModelHost.IComponentModel)Microsoft.VisualStudio.Shell.Package.GetGlobalService(typeof(Microsoft.VisualStudio.ComponentModelHost.SComponentModel));
        Workspace = componentModel.GetService<Microsoft.VisualStudio.LanguageServices.VisualStudioWorkspace>();

        View = view;
        View.LayoutChanged += OnLayoutChanged;
     }


    internal void OnLayoutChanged(object sender, Microsoft.VisualStudio.Text.Editor.TextViewLayoutChangedEventArgs e)
    {
        // Raised whenever the rendered text displayed in the ITextView changes - whenever the view does a layout
        // (which happens when DisplayTextLineContainingBufferPosition is called or in response to text or classification
        // changes), and also when the view scrolls or when its size changes.
        // Responsible for adding the adornment to any reformatted lines.

        // This code overlays the document version on line 0 of the file
        if (DocumentId == null)
        {
            var dte = Microsoft.VisualStudio.Shell.Package.GetGlobalService(typeof(EnvDTE.DTE)) as EnvDTE.DTE;
            var activeDocument = dte?.ActiveDocument; // sometimes we're constructed/invoked before ActiveDocument has been set
            if (activeDocument != null) DocumentId = Workspace.CurrentSolution.GetDocumentIdsWithFilePath(activeDocument.FullName).FirstOrDefault();
        }

        if (Adornment == null)
        {
            var line = e.NewOrReformattedLines.SingleOrDefault(l => l.Start.GetContainingLine().LineNumber == 0);
            if (line == null) return;
            var geometry = View.TextViewLines.GetMarkerGeometry(line.Extent);
            if (geometry == null) return;
            Adornment = new System.Windows.Controls.TextBlock { Width = 400, Height = geometry.Bounds.Height, Background = System.Windows.Media.Brushes.Yellow, Opacity = 0.5 };
            System.Windows.Controls.Canvas.SetLeft(Adornment, 300);
            System.Windows.Controls.Canvas.SetTop(Adornment, geometry.Bounds.Top);
            View.GetAdornmentLayer("TextAdornment1").AddAdornment(Microsoft.VisualStudio.Text.Editor.AdornmentPositioningBehavior.TextRelative, line.Extent, null, Adornment, (tag, ui) => Adornment = null);
        }

        if (DocumentId != null)
        {
            var document = Workspace.CurrentSolution.GetDocument(DocumentId);
            if (document == null) return;
            Microsoft.CodeAnalysis.VersionStamp version;
            if (!document.TryGetTextVersion(out version)) version = Microsoft.CodeAnalysis.VersionStamp.Default;
            Adornment.Text = version.ToString();
        }
    }
}

1. Project > Properties > Debug > Start External Program = "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe": when you created your VSIX project, it was set by default to launch a fresh instance of Visual Studio
2. Project > Properties > Debug > Command Line Arguments = "/rootsuffix Exp": when VS is launched, it uses an "experimental hive" whose settings and extensions don't affect your main VS instance.
3. Project > Properties > VSIX > Deploy VSIX content to experimental instance = "checked": when you start debugging, VS will install your VSIX into that experimental instance.

1. Project > Properties > Debug > Command Line Arguments = "c:\...\ConsoleApplication1.sln" /rootsuffix Exp

 

   

Example of free-form text-based data entry

Manifesto: fluid UIs for text-based data entry

• You could kick off a long-running method on Azure, which saves its state periodically. If the Azure VM goes down then another VM can just pick up where it left off.
• You could write an interactive website which prompts the user to click a button, then saves its state to disk and releases all resources. When the user clicks the button on their website, that incoming HTTP message causes the state to be picked up from where it left off.
• You could write a mobile agent, which can migrate to another machine to be near to some local resources to do the rest of its work.

1. File > New > C++ > UWP

2. References > Manage NuGet References > Add OpenCV.UWP.native.imgcodecs

This package is currently in pre-release, so it only shows up if you check the "Include Pre-release" checkbox.

It brings in two other dependencies: imgproc and imgcore.

3. Write some code! Here are some starters: add these to your #include and your using sections in MainPage.xaml.cpp:

#include <robuffer.h>
#include <opencv2\imgcodecs.hpp>
#include <opencv2\imgproc.hpp>

using namespace Windows::UI::Xaml::Media::Imaging;
using namespace Windows::Storage::Streams;
using namespace Microsoft::WRL;

Next add this inside your MainPage constructor, just after InitializeComponents:

cv::Mat bmp1 = cv::imread("demo.jpg");
cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
IBuffer^ buffer = wbmp->PixelBuffer;
unsigned char* dstPixels;
ComPtr<IBufferByteAccess> pBufferByteAccess;
ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
pBuffer.As(&pBufferByteAccess);
pBufferByteAccess->Buffer(&dstPixels);
memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
image1->Source = wbmp;

Next, add an Image to your MainPage.xaml:

<Image x:Name="image1"/>

Next find a picture, rename it "demo.jpg", and add it to your project. (By default when you drag a jpg into SolutionExplorer it gets placed in the Assets folder; the code above assumes instead that it's at top level).

4. Run it!

What's exciting is that this will run on all x86|x64|ARM configurations, and on all Debug|Release. It will pick the correct OpenCV binaries+libraries for the configuration you select.

5. Debugging: download the symbols

I tried to upload the PDBs to symbolsource.org as recommended in the NuGet docs, but symbolsource.org seems to be down or not accepting symbols. So instead, download them from here, and unzip onto your desktop:

• OpenCV.UWP.native.symbols.zip [145mb; right-click > save-as].
• Once it's downloaded, right-click > Properties > Unblock and then right-click > ExtractAll

6. Debugging: step into the OpenCV source code

• Video: Using OpenCV in a UWP C++ app [6mins]

The steps here are kind of fiddly, so that video might be the easiest way to follow them. It shows every single step explicitly.

1. The first time you debug, go to Debug > Windows > Modules, sort this list alphabetically by the Name column, look for opencv_*, right-click > LoadSymbolInformation, and point it to the symbol directory you unzipped.
2. Tools > Options > Debugging > uncheck the "Just My Code" button
3. Tools > Options > Debugging > "Enable source server support".

Once that was set up, I wanted to F11 into my call to "cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);". At first when I F11 it just stepped through header files, until it got to this line in mat.inl.hpp:

create(2, sz, _type);

This particular function isn't inline in a header file; it resides in the DLL. There's something I don't understand about C++. How come, when I F11 now it just skips over that function? And when I right-click step-into-specific then it says no source is available? but then when I hit F11 a second time then it jumps into the source code proper? Anyway, here's how to make ti work. I did right-click > Step Into Specific > cv::Mat::create. This claims the source isn't available, so I click on view disassembly, to this line:

00398F48  jmp dword ptr [__imp_cv::Mat::create (03F2490h)]

If I hit F11 a second time then it downloads the OpenCV source code file automatically from GitHub (using the "enable source server support" feature that you checked earlier, and a clever technique by Hamish Graham to use GitHub itself as the source server). You can close the disassembly now and double-click on the top frame in the callstack. Now you can step through the source code of the DLL.

 So far I've only put three of the 15 OpenCV modules on NuGet: core, imgproc and imgcodecs. The work I did to generate these NuPkgs is here on GitHub.

So what I did is write a “Roslyn Analyzer” for VB/C# and put it on NuGet. It detects whether an API I’ve used is platform-specific or version-specific. If so then it puts in a warning squiggle and gives you a quickfix (unless you've already put an ApiInformation.IsTypePresent check, or similar)

I’d love for folks to try this out and see if you find it helpful!

• To use, add a NuGet reference to PlatformSpecific.Analyzer from your VB/C# UWP project to this NuGet package (2k+ downloads so far)
• Documentation and source code here on GitHub.
• Short 3-minute training video
• More focused on the version-specific aspects, another short video
• Mike Taulty wrote a blogpost about it
• What's new in 10586 UWP? I typed up the "diff" in this file.

What I’d like to learn is this:

1. Overall, does the analyzer add value, not just for your simplistic examples, but also for your advanced/large projects?
2. The analyzer has its convention for “transitivity” where you can mark an entire method or field as platform-specific. If you use this, does it work out okay?
3. This kind of analysis can never be perfect, so it falls back on loosey-goosey heuristics. Are they too loosey-goosey? Do you find places where you’d expected a warning but it produced none? Do you find places where it produced an erroneous warning?

If people really find it useful, then I’ll push for it to be included by default in the File>New>UWP project template. But that’s a big unprecedented step so I’m cautious.

Here are some of the new 10586 APIs you could test it on! …

My goal is to figure out how to package OpenCV up into several individual NuGet packages. For instance, I expect that these files would all be packaged together into a single "OpenCV.native.flann" package. The question is, which other similar NuGet packages would it depend upon?

• DLL: opencv_flann300.dll
• import library: opencv_flann300.lib
• include file: opencv2\flann.hpp
• include files: opencv2\flann\*.h*

I looked through every DLL that makes up OpenCV and used "dumpbin" to figure out which ones depend on which others. I also looked through all the header files in the include directory to see which ones depend on which others. (I also looked at how the project files in GitHub Microsoft/opencv/tree/vs2015-samples claim to depend upon each other, but that seemed to include superfluous dependencies - ones that aren't reflected in dumpbin or in include files - so I discounted them).

The end result is this dependency graph below. This shows all the NuGet packages that I intended to create. For instance, the "OpenCV.native.flann" package will include all the files listed above, and it will have a NuGet dependency solely upon "OpenCV.native.core". A few packages also need to contain extra header files, beyond those that follow the example of "flann", and they're indicated in the graph below.

(Note: the picture above seems to have cut off the final line of headers that go into the package allheaders. It should also have included "opencv\cv.h, cv.hpp, cvaux.h*").

It's mysterious that these allheaders headers from OpenCV don't also include the headers for shape and stitching and videostab and winrt. I'm undecided whether to create an "opencv.native.complete" NuGet package that depends upon every single package, or merely the ones suggested by the "opencv.hpp" header file.

How to build OpenCV native dlls for UWP 

OpenCV is on GitHub. There are three relevant forks/branches, as of writing on November 2015:

   [1] https://github.com/itseez/opencv

     +- [2] fork https://github.com/Microsoft/opencv

          +- [3] branch https://github.com/Microsoft/opencv/tree/vs2015-samples

Repository [1] is the master. In it, the native OpenCV binaries are built for all different platforms using CMake. As far as I can tell, this repository only has support via CMake for building Windows Store and Windows Phone 8.0 and 8.1 apps. It doesn't seem to have UWP. And it also requires you to install some particular old versions of VS, as per its readme.txt.

Fork [2] doesn't seem to bring anything new.

Branch [3] contains Visual Studio .sln files from which you can build UWP versions of all the native OpenCV binaries: x86, x64 and ARM, Debug and Release. The Release builds actually were a bit wonky, but I submitted PR #69 to make them build right. So: if you clone this repository, select this branch, set an environment variable, and open up vs2015/WS/10.0/ARM/OpenCV.sln, or x86/OpenCV.sln, or x64/OpenCV.sln, then you can successfully build UWP binaries for OpenCV. This will produce 15 binaries for each architecture+build combination.

How to use OpenCV native dlls for UWP

I tried out the OpenCV native dlls in a C++ UWP app, and they worked fine.

More interestingly, the dlls all together add up to about 30mb, but I was able to use just three of them that I needed and they worked fine. This satisfies one of my central requirements, to be able to deploy just a cut-down version of OpenCV with my store app, to keep size down.

I did File>New>C++>Windows>UWP blank app. In Solution Explorer I added three DLLs as "content" files in the root directory of my project so they would be copied into the resulting appx, adjacent to the .exe:

opencv_core300d.dll
opencv_igmcodecs300d.dll
opencv_igmproc300d.dll

In Solution Explorer I added three import-libraries. By default, when you add LIBs to a C++ project, they are taken as import libraries:

opencv_core300d.lib
opencv_igmcodecs300d.lib
opencv_igmproc300d.lib

In Project Properties I added an include directory to my C++ project options

vs2015/WS/10.0/include

In my mainpage C++ file, I added a bunch of #include and using statements:

#include <robuffer.h>
#include <opencv2\imgcodecs.hpp>
#include <opencv2\imgproc.hpp>

using namespace Windows::UI::Xaml::Media::Imaging;
using namespace Windows::Storage::Streams;
using namespace Microsoft::WRL;

I added a content file "demo.jpg" in Solution Explorer, and in MainPage.xaml I added an Image control named "image1", and added these lines in OnNavigatedTo in my mainpage C++ file:

cv::Mat bmp1 = cv::imread("demo.jpg");
cv::Mat bmp2 = cv::Mat(bmp1.rows, bmp1.cols, CV_8UC4);
cv::cvtColor(bmp1, bmp2, CV_BGR2BGRA);
WriteableBitmap^ wbmp = ref new WriteableBitmap(bmp2.cols, bmp2.rows);
IBuffer^ buffer = wbmp->PixelBuffer;
unsigned char* dstPixels;
ComPtr<IBufferByteAccess> pBufferByteAccess;
ComPtr<IInspectable> pBuffer((IInspectable*)buffer);
pBuffer.As(&pBufferByteAccess);
pBufferByteAccess->Buffer(&dstPixels);
memcpy(dstPixels, bmp2.data, bmp2.step.buf[1] * bmp2.cols * bmp2.rows);
image1->Source = wbmp;

Result: it all worked. The image displayed correctly.

Next Steps

I want to package each individual DLL up as a NuGet package, for 15 NuGet packages in total. Each package should include x86+x64+ARM versions of the DLLs, and corresponding import libraries. It should also include the include files pertaining to that DLL. That way, as a C++ author, I will be able to bring in only the particular DLLs that I want. To do this I'll have to figure out the dependencies between the DLLs, and the dependencies between header files, and then express these as NuGet dependencies. (I don't yet know how to create NuGet packages for C++. I'll look to "Win2d.uwp" as the canonical example. I don't know if each package should also include debug builds, but I suspect not. I don't know where PDBs should go.)

I think that NuGet generation should be done as part of [3] branch vs2015-samples. This is the branch that has .sln files for building all the DLLs, so it's appropriate for it to also include NuGet-package-building. I don't know if this is within the vision of the owners of that branch or not. Ultimately, in the future, I don't know if the owners of Itseez/opencv would be open to their CMake system building the NuGet packages as well?

Beyond that, the major issue is that OpenCV has a C++ API. This is not friendly to pinvoke interop with .NET. OpenCV used to have a plain C API which is friendly, but the C API has been deprecated and is not getting new features. I don't wish to invent a pinvoke-friendly wrapper around the OpenCV C++ API.

Approach (1): I'd like to test the scenario of a hybrid app, where most of the app is in .NET, and it calls into a C++ winrt component that I author, and this C++ component does all the calls into OpenCV, which it obtains via NuGet.

Approach (2): OpenCVSharp has a BSD license and already contains a set of 'extern "C"' wrappers around the C++ API. The ideal would be for OpenCVSharp to be able to consume my native UWP dlls for OpenCV. There already is a "dll-free" version of OpenCVSharp, which merely contains the pinvoke wrappers, and expects you to supply the OpenCV DLLs some other way. There are a few unknowns with this approach. I don't know if the OpenCVSharp project has different version of OpenCV (and hence different include files) as compared to the one in branch [3]. I don't know if the native extern "C" assembly that comes with OpenCVSharp will be compatible with the DLLs I build, and the fact that there are 15 of them rather than just 1 big "world" dll. I don't know if it will need to switch to "delay-load" to support the scenario where you only package say 3 of the OpenCV dlls with your store app.

Notes

I am hugely excited! It was thrilling to see the UWP versions of all the DLLs get built properly. I think I know how to package up DLLs into NuGet better than most folk, and I can do a best-in-class job. I'm excited that I was able to get something working end-to-end at least in C++.

I want to use it from my .NET UWP apps, but I'm struggling to figure out how. If anyone has solid guidance on how to proceed, I'd love to know! This blog post is just a documentation of my first tentative investigations. (Also eventually I want to use OpenCV from my cross-platform .NET Core apps as well, so I can write an ASP.NET5 webservice running on Linux that does face detection).

I think there should ideally be three layers of NuGet packages:

1. OpenCV.dlls - there should just be a base NuGet package whose only job is to carry the native C++ dlls. It needs to carry the three flavors, x86 + x64 + ARM. It should use the new features in NuGet3 so it can carry all three flavors of dll in a single NuGet package, and at build-time the project.json should pick out the right binaries.
• There's a catch. The normal "opencv_world300.dll" is huge at 30mb and includes absolutely all OpenCV functionality. Typically my store apps will only need a small subset of that functionality. I see that the "OpenCV.NET" NuGet package instead delivers it split up into 19 separate DLLs, each of them about 1-2mb. It would be best if you could get the DLLs also through a load of smaller NuGet packages, one by one, to pick out which of those binaries you need.
• Note that a C++ DLL that you built for an earlier version of Windows Store or Windows Phone will not work in UWP. That's because even though UWP supports all the same APIs that were there in the earlier WinRT platforms, it divides them up into different "api-sets", so the binaries are incompatible.
• Ideally this NuGet package should also carry the native C++ dlls for Linux and for OSX, so it can be used on other .NET Core platforms. I don't know how native dlls are carried on those platforms
   
2. OpenCV.interop - there should be an intermediate NuGet package whose job is solely to contain p/invoke declarations for those DLLs without any fancy wrapping.
• Maybe it would also include header files, for consumption by C++
• I don't know how p/invoke declarations work for cross-platform .NET Core.
• Why decouple this from the raw DLLs? solely to enable the "pick-and-choose" idea above. Maybe it's not worth it. Maybe the "OpenCV.dlls" and "OpenCV.interop" should be combined.
   
3. OpenCV.Net - there should be a higher-level set of wrapper classes, more .NET friendly.
• Maybe this package need never be written, if everyone is happy using the unwrapped versions of the APIs.
• Maybe there will be multiple competing versions of this package, with different API designs, some more friendly than others.

OpenCV itself 

http://opencv.org/ - the main website has a "download binaries for Windows" link. This includes x64+x86 builds of "opencv_world300.dll" (32mb). I don't know if they're built for UWP but I suspect not. It has two 8.1-era Phone+Store sample apps but they didn't build out of the box. It has a directory called "sources\platforms\winrt" which has instructions for building on WinRT, but these instructions don't include x64, and I suspect they date from before UWP.

https://github.com/Itseez/opencv - the main GitHub repository for OpenCV

https://msopentech.com/blog/2015/03/06/opencv-enabled-for-windows-runtime-family/ - has pre-UWP instructions on how to build binaries from the main GitHub repository using CMake

https://www.nuget.org/packages/OpenCV/ - this NuGet package with 18k downloads claims to contain "binaries and build targets". This isn't true. It contains C++ header files, and it contains a .targets file which (based on build parameters) pick up the OpenCV native DLLs that it assumes you got from elsewhere and copied into a particular location. It only does this for x86 and x64; not ARM.

There are two old sites - https://github.com/Microsoft/opencv was the old staging repo where WinRT support was first added to the main OpenCV GitHub, and http://code.opencv.org/projects/opencv/wiki/WindowsRT had old instructions on how to build for WinRT. I think both links are out of date.

Analysis: I suspect there are no "ready-to-go" binaries here for UWP, nor any useful NuGet packages. If I want to use OpenCV from source, I'll have to learn the thing's build system, configure it to build UWP, and built it myself. I'll have to write a OpenCV.dlls NuGet package from scratch myself. I don't know whether the work I do here can be submitted back into OpenCV's daily builds, or if I'll have to keep it separate.

OpenCV Extension SDK 

https://channel9.msdn.com/Events/Build/2015/3-82 - this video discusses an "OpenCV for Windows Store" Extension SDK which contains native binaries for UWP apps. However, the Extension SDK has never been publically released. (Also, ExtensionSDKs were used in the past because prior versions of NuGet weren't good at selecting x64/x86/ARM binaries based on build-time configuration. That limitation has gone away, and now NuGet is a much better option than ExtensionSDK).

Analysis: The author of the video has moved on to a different team. I've contacted his former colleagues to see if I can get anywhere with this. I don't know if it will be a single huge 30mb "world" dll, or a series of smaller ones.

OpenCVSharp

This NuGet package contains a 30mb "world" binary for x86+x64 (not ARM) that come from here. It also contains C# wrappers that are as close as possible to the raw C API. This is the most widely used package, with 32k downloads total.

I'm confused, because the NuGet package has that huge DLL, but then the project's install.ps1 seems to have them split up into separate pieces.

The NuGet package only installs onto desktop .NET Framework 4.0 and higher -- not .NETCore, and not UWP.

As for the raw native binaries themselves, I don't know. But given the desktop focus, and the lack of ARM, I strongly suspect they won't work on UWP.

https://github.com/shimat/opencvsharp - this is the main repository.

The OpenCVSharp NuGet is actually offered in four flavors:

1. https://www.nuget.org/packages/OpenCvSharp3-AnyCPU/ - the main one for OpenCV3 which was released in 2015; most recent
2. https://www.nuget.org/packages/OpenCvSharp3-WithoutDll/ - as above, but missing the "world.dll" (so you'll pick it up yourself elsewhere)
3. https://www.nuget.org/packages/OpenCvSharp-AnyCPU/ - like [1] but for OpenCV 2.4
4. https://www.nuget.org/packages/OpenCvSharp-WithoutDll/ - like [2] but for OpenCV 2.4

There's also this NuGet package https://www.nuget.org/packages/OpenCVSharp/ but it doesn't seem to be related.

Analysis: this project seems to provide the correct "interop p/invoke" layer. But I think it won't be a good source of binaries for UWP.

OpenCV.NET

This NuGet package has a nicer .NET-style set of wrapper classes. I don't understand how/where the NuGet package picks up its raw binaries from, but it appears to get them somehow! The binaries seem to be x86 only (not x64, not ARM) and are split into ~15 smaller DLLs. The package has been downloaded 8k times. The most recent update was 9 months ago.

The NuGet package only works on desktop .NET Framework 4.0 and higher (not UWP).

https://www.nuget.org/packages/OpenCV.Net/

Analysis: I think it'd be ideal to split out the "interop p/inoke" layer from the wrapper classes. I'm interested to know whether the interop declarations for "split-into-separate-dlls" that this project uses is different from the interop declarations for "world.dll". In any case, this project isn't a suitable source for raw binaries.

Emgu

This project does provide ARM and WinRT for 8.1-era. However I haven't read anything to indicate support for UWP. Its NuGet package was last updated 20 months ago and has 6k downloads.

This project requires a commercial license for ARM and WinRT.

http://www.emgu.com/wiki/index.php/Main_Page

https://www.nuget.org/packages/VVVV.EmguCV/

Analysis: this isn't in the OSS spirit that I like about .NET Core, so I haven't pursued it.

Stack Overflow 

http://stackoverflow.com/questions/14065370/using-opencv-in-a-windows-store-app-with-visual-c - this seems to be the most valuable StackOverflow resource. It has Raman Sharma (author of the "OpenCV for Windows Store Apps" ExtensionSDK) give some answers on how he built OpenCV for WinRT.

I've put this functionality into a NuGet package. To use UWP APIs from your >=.NET45 desktop app, simply add a NuGet reference:

• UWP for Desktop NuGet package

I've also moved the technical information from this blog over to the package's GitHub project site:

• UWP for Desktop GitHub project page

Out of date: first steps

Add these references to your project, by right-clicking on your project in Solution Explorer and doing References > AddReference > Browse. This minimum needed to get going.

1. c:\Program Files (x86)\Windows Kits\10\References\Windows.Foundation.FoundationContract\1.0.0.0\Windows.Foundation.FoundationContract.winmd
2. c:\Program Files (x86)\Windows Kits\10\References\Windows.Foundation.UniversalApiContract\1.0.0.0\Windows.Foundation.UniversalApiContract.winmd
3. c:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD
4. c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5\System.Runtime.WindowsRuntime.dll

The complete set of Desktop WinRT APIs

Here is a MSDN list of WinRT APIs that can be called from desktop apps (but I believe this list is incomplete).

Just the four references is enough to get you 90% of the WinRT APIs you'll need in practice.

But to go the full mile, I think the best way would be to package it up as a NuGet package according to the instructions below. I'll explore that idea later. I'd want to get the other UWP facades, get something that works with newer UWP SDKs than just 10240, also include APIs from the Desktop Extension SDK, include the remaining .NET interop DLLs, and get all the default namespace-imports.

Until I or someone else produces that NuGet package, here below is a detailed explanation:

Explanation

To find the UWP WinRT APIs themselves, look for example at this file:

C:\Program Files (x86)\Windows Kits\10\Platforms\UAP\10.0.10240.0\Platform.xml

This Platform.xml file conveys which WinRT APIs are available in "version 10240" of the UWP SDK. (This was the version that came out at VS2015 RTM. Newer versions will come out periodically). The Platform.xml file works by being an index into a set of other winmd files. It looks like this:

<ApiContract name="Windows.Foundation.FoundationContract" version="1.0.0.0" />
<ApiContract name="Windows.Foundation.UniversalApiContract" version="1.0.0.0" />
<ApiContract name="Windows.Networking.Connectivity.WwanContract" version="1.0.0.0" />

Each line denote a .winmd file in the "C:\Program Files (x86)\Windows Kits\10\References directory". That's how I picked up exact filenames and pathnames to add as references. (I should really have added a reference to the WwanContract.winmd file as well, but couldn't be bothered)

These three winmd files make up the WinRT APIs available across all Win10 devices. You might additionally want to use the additional WinRT APIs that are part of Desktop. Look at this file:

C:\Program Files (x86)\Windows Kits\10\Extension SDKs\WindowsDesktop\10.0.10240.0\SDKManifest.xml

It has a load more <ApiContract> directives, pointing to additional .winmd files you should reference from the "C:\Program Files (x86)\Windows Kits\10\References" directory. I haven't bothered to do that.

To interop with DLLs or WINMDs build for Win8.1 or WindowsPhone8.1, the issue is that those old DLLs and WinMDs used to think that all WinRT types were in a file called "windows.winmd". But they're not anymore! Instead they're in files called Windows.Foundation.FoundationContract.winmd and the like. The solution is a façade winmd, which is basically a load of type-forwarders from windows.winmd to the new winmd files.

c:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD

To use "await" on WinRT types, this and other interop between .NET and WinRT are provided by the following DLL. Note that it's an 8.1-era DLL, and therefore only works in conjunction with "facade\windows.winmd" above. (There are several other interop assemblies but they're not so crucial).

c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5\System.Runtime.WindowsRuntime.dll

If you fail to add both these two assemblies, then you'll get error messages like this:

• BC36930: 'Await' requires that the type 'IAsyncAction' have a suitable GetAwaiter method
• CS0012: The type 'IAsyncAction' is defined in an assembly that is not referenced. You must add a reference to assembly 'Windows, Version=255.255.255.0, Culture=neutral, PublicKeyToken=null, ContentType=WindowsRuntime'.
• CS4036: 'IAsyncAction' does not contain a definition for 'GetAwaiter' and no extension method 'GetAwaiter' accept a first argument of type 'IAsyncAction' could be found (are you missing a using directive for 'System'?)

Example code (VB)

Here's an example console app in VB which uses UWP WinRT APIs:

Imports System.IO

Module Module1
    Sub Main()
        MainAsync().GetAwaiter().GetResult()
    End Sub

    Async Function MainAsync() As Task
        ' basics
        Dim folder = Windows.Storage.KnownFolders.DocumentsLibrary
        Dim opts As New Windows.Storage.Search.QueryOptions(
                              Windows.Storage.Search.CommonFileQuery.OrderByName, {".txt"})
        Dim files = Await folder.CreateFileQueryWithOptions(opts).GetFilesAsync(0, 20)
        For Each file In files
            Console.WriteLine(IO.Path.GetFileName(file.Path))
        Next

        ' streams
        Using reader = New IO.StreamReader(Await files.First.OpenStreamForReadAsync())
            Dim txt = Await reader.ReadToEndAsync()
            Console.WriteLine(txt.Substring(0, 100))
        End Using

        ' pictures
        Dim pics = Await Windows.Storage.KnownFolders.PicturesLibrary.GetFilesAsync(
                         Windows.Storage.Search.CommonFileQuery.OrderBySearchRank, 0, 1)
        Using stream = Await pics.First.OpenReadAsync()
            Dim decoder = Await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream)
            Console.WriteLine(decoder.OrientedPixelWidth & "x" & decoder.OrientedPixelHeight)
        End Using

        ' httpclient
        Dim client As New Net.Http.HttpClient()
        Dim html = Await client.GetStringAsync("http://www.microsoft.com")
        Console.WriteLine(html.Substring(0, 100))

    End Function
End Module

Example code (C#)

Here's an example console app in C# which uses UWP WinRT APIs:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        MainAsync().GetAwaiter().GetResult();
    }

    static async Task MainAsync()
    {
        // basics
        var folder = Windows.Storage.KnownFolders.DocumentsLibrary;
        var opts = new Windows.Storage.Search.QueryOptions(Windows.Storage.Search.CommonFileQuery.OrderByName, new[] { ".txt" });
        var files = await folder.CreateFileQueryWithOptions(opts).GetFilesAsync(0, 20);
        foreach (var file in files)
        {
            Console.WriteLine(System.IO.Path.GetFileName(file.Path));
        }

        // streams
        using (var reader = new System.IO.StreamReader(await files.First().OpenStreamForReadAsync()))
        {
            var txt = await reader.ReadToEndAsync();
            Console.WriteLine(txt.Substring(0, 100));
        }

        // pictures
        var pics = await Windows.Storage.KnownFolders.PicturesLibrary.GetFilesAsync(
                               Windows.Storage.Search.CommonFileQuery.OrderBySearchRank, 0, 1);
        using (var stream = await pics.First().OpenReadAsync())
        {
            var decoder = await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream);
            Console.WriteLine($"{decoder.OrientedPixelWidth} x  {decoder.OrientedPixelHeight}");
        }

        // httpclient
        var client = new System.Net.Http.HttpClient();
        var html = await client.GetStringAsync("http://www.microsoft.com");
        Console.WriteLine(html.Substring(0, 100));
    }
}

Motivation

• Your .vbproj/.csproj no longer gets polluted with NuGet artifacts. They are kept completely separate.
• You can change your app target platform, and change debug/release and x86/x64/arm/anycpu, and NuGet packages will now pick this up immediately -- before you had to uninstall+reinstall them.
• You can have two different solutions in two different directories that include the same project (this is particularly useful when you're working across two different repositories)
• Your SolutionExplorer>References node looks cleaner because it only includes the packages you've actually installed, rather than all their dependencies too.
• Uninstalling NuGet packages is easier, again because you only have to uninstall the ones you installed rather than all their dependencies.
• Packages are cached globally (on a per-user per-machine) basis rather than being downloaded and unzipped into every single solution that uses them.
• File > New and Managed NuGet Packages > Install both become faster.
• You get more precise control over NuGet package upgrades, and version mismatches.

• In ASP.NET 5 ("DNX") projects
  • Currently still in preview form
• For Windows 10 Universal app projects ("UWP")
  • Requires installation of the VS Universal Windows app development tools. If they're not already installed, you can install them with "Add/Remove Programs > VisualStudio2015 > Modify".
• For modern Portable Class Library ("PCL") development
  • "Modern PCLs" are ones whose targets are limited to amongst ASP.NET Core 5.0 ("DNX"), Windows 10 ("UWP") and .NET46.
  • Again this requires installation of the VS universal windows tools.
• For any other project type,
  • You can delete packages.config and add a project.json file manually, to take advantage of its benefits [instructions].
  • This path isn't officially supported. It mostly works, except that if you have a P2P (Project-to-project) reference from a packages.config project to a project.json library. In that case, any NuGet dependencies in project.json will fail to be copied to the consuming project.
  • VS universal windows tools are required.

Master table of all the various .NET frameworks

1. As a NuGet package author you have to be aware of the different versions to write your NuGet packages effectively.
2. Microsoft has gotten on top of the problem with a solution called the "dotnet" TxM, described below (TxM stands for "Target x Moniker" and refers to the codes in the table below). The project.json approach was introduced in part to support dotnet.

• .NET46. This is the framework that's installed as part of the Windows Desktop+Server operating system. If you write an app which targets this, and you add a reference to the System.Collections NuGet package, all it says is " nothing to do here since System.Collections is already part of the OS platform which this code will run on". 
• UAP10.0 (Windows 10 Universal Apps) and DNXCORE50 (ASP.Net 5 Core). These platforms expect apps to carry their .NET runtime and framework with them, app-locally. If you write an app which targets either of these platforms, and you add a reference to the System.Collections NuGet package, then it copies the actual DLL which implements List<T> right there into your application's "bin" directory, adjacent to your EXE.

A worked example

Step 1: Create the DLL

• You can't P2P reference your Profile259 PCL from a Modern PCL that has ASP.NET Core checked. (A "Profile259 PCL" is one whose targets are as picked at the top of this section; a Modern PCL is one whose targets are limited to amongst .NET46, Win10, ASP.NET Core). This is a point-in-time bug that will be fixed in the first VS2015 update. As a workaround you can still do a raw file reference. And once you turn your PCL into a NuGet package then it will work fine. Another workaround, for testing purposes, is to just go to your consuming project's targets and remove ASP.NET Core and then the P2P reference will work.
• You can't P2P reference your Profile259 PCL from a "DNX" project (i.e. ASP.Net 5). And you can't even add a raw file reference because the "Browse" button doesn't work in DNX projects. Don't worry. Once you turn your PCL into a NuGet package then it will work fine.

Step 2: plan the NuPkg directory structure

WorkedExample.1.0.0.0.nupkg +-- WorkedExample.nuspec +-- lib +-- dotnet |    +-- ClassLibrary1.dll +-- portable-net45+win8+wpa81+wp8 +-- ClassLibrary1.dll

• If your project isn't a portable class library, then it uses these tie-breakers in order to search for the best match…
  1. Look for an "exact match platform". For instance, if your project is "uap10.0" and there is a subdirectory called "lib\uap10.0" then use it.
  2. Failing that, look for an earlier version of the platform or for the unnumbered code for that platform (i.e. in the Master Table, look in the same row, but further to the left)
     • Example: if the project is .NET45, and there's no "lib\net45" subdirectory, then look for "lib\net40", failing that "lib\net35" and so on.
     • Example: if the project is UWP and there's no "lib\uap10.0" subdirectory, then look for "lib\uap", failing that "lib\win81" or "lib\netcore451", and so on.
  3. Failing that, and if your project is >=45, then look for "lib\dotnet"
  4. Failing that, look for all subdirectories of the form "lib\portable-xyz" where your project type is amongst those "xyz" as per rules 1..4. Amongst all of those portable-* candidates, there's a rule for picking the best one, but I don't know it.
  5. Failing that, packages.config used to look just in the root "lib\" folder, but that functionality is no longer supported in packages.json. If your nupkg used to put DLLs directly in the root "lib\" folder then it has to change.
• If your project is a PCL,
  1. If your project is a "modern PCL" and there exists a subdirectory "lib\dotnet", then use it. (A "modern PCL" is one which targets some or all of .NET46, Win10 and ASP.NetCore, and which targets nothing other than those three.)
  2. Failing that, calculate your project's targets. In the case of a traditional PCL, the targets are those listed in the PCL targets dialog. In the case of a modern PCL, I don't know what the targets are. Then look for a lib subdirectory that is an exact match for your project's targets.
  3. Failing that, look for the best-match "lib\portable-*" subdirectory. I don't know the rules for "best-match" in this case.

Step 3: Craft the .nuspec dependencies

<?xml version="1.0"?> <package> <metadata> <id>SampleNuGetPackage</id> <version>1.0.0.2</version> <title>Sample NuGet Package</title> <authors>Lucian Wischik</authors> <owners>Lucian Wischik</owners> <requireLicenseAcceptance>false</requireLicenseAcceptance> <description>This is a demo portable .NET class library</description> <summary>It is referencable by all >=.NET45 clients, including WindowsStore and Phone</summary> <dependencies> <group targetFramework ="net"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="win"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="wp"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="wpa"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="MonoAndroid"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="MonoTouch"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="Xamarin.iOS"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="portable-net45+win8+wpa81+wp8"> <dependency id="Newtonsoft.Json" version="7.0.1" /> </group> <group targetFramework ="uap"> <dependency id="Newtonsoft.Json" version="7.0.1" /> <dependency id="System.Runtime" version="4.0.0"/> <dependency id="System.Resources.ResourceManager" version="4.0.0"/> <dependency id="System.Collections" version="4.0.0"/> <dependency id="System.Linq" version="4.0.0"/> </group> <group targetFramework ="dotnet"> <dependency id="Newtonsoft.Json" version="7.0.1" /> <dependency id="System.Runtime" version="4.0.0"/> <dependency id="System.Resources.ResourceManager" version="4.0.0"/> <dependency id="System.Collections" version="4.0.0"/> <dependency id="System.Linq" version="4.0.0"/> </group> </dependencies> </metadata> </package>

Public Class Class1 Public Shared Function f() As String     Dim s = "[""hello"",""world""]" Dim x = Newtonsoft.Json.JsonConvert.DeserializeObject(Of List(Of String))(s) Return $"{x.First} I am here" End Function

End Class

<?xml version="1.0" encoding="utf-8"?> <packages> <package id="Newtonsoft.Json" version="7.0.1" /> </packages>

.assembly extern System.Runtime {.ver 4:0:0:0 }

.assembly extern System.Resources.ResourceManager { .ver 4:0:0:0 }

.assembly extern System.Collections { .ver 4:0:0:0 }

.assembly extern Newtonsoft.Json { .ver 7:0:0:0 }

.assembly extern System.Linq { .ver 4:0:0:0 }

1. Newtonsoft.Json version 7.0.1
   1. This comes straight from my library project's "packages.config"
   2. Note that 7.0.1 is the version of the Newtonsoft.Json NuGet package which I depend upon and I read this version number from packages.config; the number { .ver 7:0:0:0 } doesn't refer to a NuGet package version; it is instead the assembly version of the file Newtonsoft.Json.dll which is contained within that NuGet package.
   3. It's good to write the exact NuGet package version number you depend upon (rather than just writing an earlier number like 7.0.0). This way, NuGet package restore can avoid an extra search.
2. System.Runtime version 4.0.0 System.Resources.ResourceManager version 4.0.0 System.Collections version 4.0.0 System.Linq version 4.0.0
   1. For the .NET framework, Microsoft adopted a simple 1:1 mapping between DLL names, and the NuGet packages that contain them.
   2. (This 1:1 mapping only works because the PCL targets none of the red platforms in the Master Table. Those red platforms simply use mscorlib as a reference, which wouldn't provide any clues as to which NuGet packages are needed).
   3. I had no idea which NuGet package versions to use. So I just did File>New>UWP as a throwaway example of a modern platform, built it, then opened the "project.lock.json" file. From this I was able to find some typical NuGet package version numbers for each of these NuGet packages.

• [Full dependencies] Newtonsoft.Json, System.Runtime, System.Resources.ResourceManager, System.Collections, System.Linq

• [Brief dependencies] Newtonsoft.Json
• (Pedantically, you were supposed to add <frameworkAssembly> directives to your .nuspec just in case the user had manually deleted important framework references from their References node. But no one ever used these directives correctly, and it wasn't worth doing.)

1. I might omit the <dependencies> section entirely. But then if my NuGet package is referenced by a project which doesn't also reference Newtonsoft.Json, then it will fail at runtime with a DllLoadException.
2. I might merely put into <dependencies> the items in my library project's packages.config, in this case Newtonsoft.Json. This will run fine on all traditional platforms. It will likely run fine on uap10.0 unless the app author has gone out of their way to delete the default NuGet references, or unless Microsoft in future decides to trim down the set of default NuGet references for a faster leaner system. It will likely fail on dnxcore50, since those projects have just a tiny set of default NuGet references. It will likely fail on future platforms, since they're most likely to follow the lead of dnxcore50.
3. I might merely put into the <dependencies> those items in my library project's packages.config, plus the single catch-all "Microsoft.NETCore" . The Microsoft.NETCore package represents all of the default NuGet references. This approach will run fine on uap10. It will run fine on dnxcore50 and on future platforms, but it will bloat them due to including "the whole world and the kitchen sink" while they aim to be lean. It will work on traditional platforms, but will look really terrible since 100+ NuGet references will be added to them.
4. I might abandon the complicated "dependency groups", and merely use a flat list of dependencies . As discussed above, this will work on all platforms. On traditional platforms, however, it adds superfluous NuGet references to traditional platforms -- in this case just 4 superfluous references plus their transitive dependencies, but in more realistic libraries it might add 50 or more.

Step 4: produce the .nupkg file as part of your build

Special Circumstances