Windows.Media.Ocr scan by column and not by row

Mattia Fanti 356 Reputation points
2022-01-06T15:29:30.107+00:00

In windows form .net framework 4.8 I'm using Windows.Media.Ocr to scan some text, but instead to scan by row it scans by column.
For example, if I try to scan
https://imgur.com/wf005mB

the ocr output is:

0.00003829 0.00003821 0.00003821 0.00003821 0.00003830 0.00003829 236.1K 552.7K 575.4K 306.9K 77K 402K

Instead of:

0.00003829 236.1K

0.00003821 552.7K
0.00003821 575.4K
0.00003821 306.9K
0.00003830 77K
0.00003829 402K

The code I'm using is taken from https://learn.microsoft.com/en-us/answers/questions/680872/how-to-parse-numbers-from-screen.html

Imports Windows.Media.Ocr  
        Imports System.IO  
        Imports System.Runtime.InteropServices.WindowsRuntime  
        Public Class Form1  

            Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click  
                Dim softwareBmp As Windows.Graphics.Imaging.SoftwareBitmap  
                Using bmp As Bitmap = New Bitmap(PictureBox1.Width, PictureBox1.Height)  
                    Using g As Graphics = Graphics.FromImage(bmp)  
                        Dim pt As Point = Me.PointToScreen(New Point(PictureBox1.Left, PictureBox1.Top))  
                        g.CopyFromScreen(pt.X, pt.Y, 0, 0, bmp.Size, CopyPixelOperation.SourceCopy)  
                        Using memStream = New Windows.Storage.Streams.InMemoryRandomAccessStream()  
                            bmp.Save(memStream.AsStream(), System.Drawing.Imaging.ImageFormat.Bmp)  
                            Dim decoder As Windows.Graphics.Imaging.BitmapDecoder = Await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(memStream)  
                            softwareBmp = Await decoder.GetSoftwareBitmapAsync()  
                        End Using  
                    End Using  
                End Using  

                Dim ocrEng = OcrEngine.TryCreateFromLanguage(New Windows.Globalization.Language("en-US"))  

                Dim languages As IReadOnlyList(Of Windows.Globalization.Language) = ocrEng.AvailableRecognizerLanguages  
                For Each language In languages  
                    Console.WriteLine(language.LanguageTag)  
                Next  
                Dim r = ocrEng.RecognizerLanguage  
                Dim n = ocrEng.MaxImageDimension  
                Dim ocrResult = Await ocrEng.RecognizeAsync(softwareBmp)  
                RichTextBox1.Text = ocrResult.Text  
            End Sub  
        End Class  

under suggestion of Castorix, "Each word has BoundingRect, so you could add them in a List and sort them from Bottom and Left" but I literally don't know how to implement it. I've tried different examples online but It seems really above my capacities and I can't really find any topic linked to mine.
Any help would be appreciated.
Thanks

Developer technologies | VB
{count} votes

Accepted answer
  1. Castorix31 90,681 Reputation points
    2022-01-06T16:45:10.257+00:00

    I did a test, which could be improved.
    Updated end of code (from Dim ocrResult =...), by extracting words (I rounded Bottom for letters which are not aligned, but not perfect...),
    then by sorting them by Bottom, then Left :

            Dim ocrResult = Await ocrEng.RecognizeAsync(softwareBmp)
    
            RichTextBox1.Clear()
    
            Dim wordList As List(Of cText) = New List(Of cText)()
            Dim lines As IReadOnlyList(Of OcrLine) = ocrResult.Lines
            For Each line In lines
                For Each word In line.Words
                    Dim nY As Double = CLng(word.BoundingRect.Bottom / 10) * 10
                    wordList.Add(New cText() With {.Text = word.Text, .LocY = nY, .LocX = word.BoundingRect.Left})
                Next
            Next    
    
            wordList.Sort(New WordComparer())
    
            Dim oldLocY As Double = 0
            For Each item As cText In wordList
                'Console.WriteLine(item.Text)
                If (item.LocY > oldLocY And oldLocY <> 0) Then
                    RichTextBox1.Text += Environment.NewLine
                End If
                RichTextBox1.Text += item.Text + " "
                oldLocY = item.LocY
            Next
    

    with :

        Public Class cText
            Public Property Text As String
            Public Property LocY As Double
            Public Property LocX As Double
        End Class
    
    
        Class WordComparer
            Implements IComparer(Of cText)
            Public Function Compare(p1 As cText, p2 As cText) As Integer Implements System.Collections.Generic.IComparer(Of cText).Compare
                Dim result = p1.LocY.CompareTo(p2.LocY)
                If result = 0 Then
                    result = p1.LocX.CompareTo(p2.LocX)
                End If
                Return result
            End Function
        End Class
    
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.