Szerkesztés

Megosztás a következőn keresztül:

Text Recognition in the Windows App SDK

  • Cikk

The new Artificial Intelligence (AI) text recognition APIs that will ship with Windows App SDK 1.6 Experimental 2 can be used to identify characters in an image, recognize words, lines, polygonal boundaries, and provide confidence levels for the generated matches.

These new Windows App SDK APIs are faster and more accurate than the legacy Windows.Media.Ocr.OcrEngine APIs in the Windows platform SDK and support hardware acceleration in devices with a neural processing unit (NPU).

Important

The experimental channel includes APIs and features in early stages of development. All APIs in the experimental channel are subject to extensive revisions and breaking changes and may be removed from subsequent releases at any time. They are not supported for use in production environments, and apps that use experimental features cannot be published to the Microsoft Store.

Prerequisites

What can I do with the Windows App SDK and AI Text Recognition?

Use the AI Text Recognition features shipping with Windows App SDK 1.6 Experimental 2 to identify and recognize text in an image. You can also get the text boundaries and confidence scores for the recognized text.

Create an ImageBuffer from a file

In this example we call a LoadImageBufferFromFileAsync function to get an ImageBuffer from an image file.

In the LoadImageBufferFromFileAsync function, we complete the following steps:

  1. Create a StorageFile object from the specified file path.
  2. Open a stream on the StorageFile using OpenAsync.
  3. Create a BitmapDecoder for the stream.
  4. Call GetSoftwareBitmapAsync on the bitmap decoder to get a SoftwareBitmap object.
  5. Return an image buffer from CreateBufferAttachedToBitmap.
namespace winrt
{
    using namespace Microsoft::Windows::AI::Imaging;
    using namespace Windows::Graphics::Imaging;
    using namespace Windows::Storage;
    using namespace Windows::Storage::Streams;
}

winrt::IAsyncOperation<winrt::ImageBuffer> LoadImageBufferFromFileAsync(
    const std::wstring& filePath)
{
    auto file = co_await winrt::StorageFile::GetFileFromPathAsync(filePath);
    auto stream = co_await file.OpenAsync(winrt::FileAccessMode::Read);
    auto decoder = co_await winrt::BitmapDecoder::CreateAsync(stream);
    auto bitmap = co_await decoder.GetSoftwareBitmapAsync();
    if (bitmap == nullptr) {
        co_return nullptr;
    }
    co_return winrt::ImageBuffer::CreateBufferAttachedToBitmap(bitmap);
}

Recognize text in a bitmap image

The following example shows how to recognize some text in a SoftwareBitmap object as a single string value:

  1. Create a TextRecognizer object through a call to the EnsureModelIsReady function, which also confirms there is a language model present on the system.
  2. Using the bitmap obtained in the previous snippet, we call the RecognizeTextFromSoftwareBitmap function.
  3. Call CreateBufferAttachedToBitmap on the image file to get an ImageBuffer object.
  4. Call RecognizeTextFromImage to get the recognized text from the ImageBuffer.
  5. Create a wstringstream object and load it with the recognized text.
  6. Return the string.

Note

The EnsureModelIsReady function is used to check the readiness state of the text recognition model (and install it if necessary).

namespace winrt
{
    using namespace Microsoft::Windows::Imaging;
    using namespace Microsoft::Windows::Vision;
    using namespace Windows::Graphics::Imaging;
}

winrt::IAsyncOperation<winrt::TextRecognizer> EnsureModelIsReady();

winrt::IAsyncOperation<winrt::hstring> RecognizeTextFromSoftwareBitmap(winrt::SoftwareBitmap const& bitmap)
{
    winrt::TextRecognizer textRecognizer = co_await EnsureModelIsReady();
    winrt::ImageBuffer imageBuffer = winrt::ImageBuffer::CreateBufferAttachedToBitmap(bitmap);
    winrt::RecognizedText recognizedText = textRecognizer.RecognizeTextFromImage(imageBuffer);
    std::wstringstream stringStream;
    for (const auto& line : recognizedText.Lines())
    {
        stringStream << line.Text().c_str() << std::endl;
    }
    co_return winrt::hstring{stringStream.view()};
}

winrt::IAsyncOperation<winrt::TextRecognizer> EnsureModelIsReady()
{
  if (!winrt::TextRecognizer::IsAvailable())
  {
    auto loadResult = co_await winrt::TextRecognizer::MakeAvailableAsync();
    if (loadResult.Status() != winrt::PackageDeploymentStatus::CompletedSuccess)
    {
        throw winrt::hresult_error(loadResult.ExtendedError());
    }
  }

  co_return winrt::TextRecognizer::CreateAsync();
}

Get word bounds and confidence

Here we show how to visualize the BoundingBox of each word in a SoftwareBitmap object as a collection of color-coded polygons on a Grid element.

Note

For this example we assume a TextRecognizer object has already been created and passed in to the function.

namespace winrt
{
    using namespace Microsoft::Windows::Imaging;
    using namespace Microsoft::Windows::Vision;
    using namespace Micrsooft::Windows::UI::Xaml::Controls;
    using namespace Micrsooft::Windows::UI::Xaml::Media;
    using namespace Micrsooft::Windows::UI::Xaml::Shapes;
}

void VisualizeWordBoundariesOnGrid(
    winrt::SoftwareBitmap const& bitmap,
    winrt::Grid const& grid,
    winrt::TextRecognizer const& textRecognizer)
{
    winrt::ImageBuffer imageBuffer = winrt::ImageBuffer::CreateBufferAttachedToBitmap(bitmap);
    
    winrt::RecognizedText result = textRecognizer.RecognizeTextFromImage(imageBuffer);

    auto greenBrush = winrt::SolidColorBrush(winrt::Microsoft::UI::Colors::Green);
    auto yellowBrush = winrt::SolidColorBrush(winrt::Microsoft::UI::Colors::Yellow);
    auto redBrush = winrt::SolidColorBrush(winrt::Microsoft::UI::Colors::Red);
    
    for (const auto& line : recognizedText.Lines())
    {
        for (const auto& word : line.Words())
        {
            winrt::PointCollection points;
            const auto& bounds = word.BoundingBox();
            points.Append(bounds.TopLeft);
            points.Append(bounds.TopRight);
            points.Append(bounds.BottomRight);
            points.Append(bounds.BottomLeft);

            winrt::Polygon polygon;
            polygon.Points(points);
            polygon.StrokeThickness(2);

            if (word.Confidence() < 0.33)
            {
                polygon.Stroke(redBrush);
            }
            else if (word.Confidence() < 0.67)
            {
                polygon.Stroke(yellowBrush);
            }
            else
            {
                polygon.Stroke(greenBrush);
            }

            grid.Children().Add(polygon);
        }
    }
}

Additional resources

Access files and folders with Windows App SDK and WinRT APIs

Related content