How To Move Mouse Cursor to Specific Image based on Saved Image

Valleze 1 Reputation point
2022-03-27T22:34:50.65+00:00

I have a button image saved as png image.

I'm trying to create some autoclick in visual studio 2022 to click in this button ever it appears on my screen. But it appears in diferent positions and diferent times on screen.

I'm using c# language

I'm using winforms

saved button image is equal the real button image

I don't know about image recognition, but I know it's possible.

Windows Forms
Windows Forms
A set of .NET Framework managed libraries for developing graphical user interfaces.
1,903 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Jack J Jun 24,516 Reputation points Microsoft Vendor
    2022-03-28T08:50:17.627+00:00

    @Valleze , Welcome to Microsoft Q&A, based on my research, I find a solution to do it.

    Please install following nuget-package:

    187453-image.png

    Then you could try the following code to move mouse cursor to button location.

    Code:

     public int X { get; set; }  
            public int Y { get; set; }  
            protected override bool ProcessCmdKey(ref Message msg, Keys keyData)  
            {  
                if(keyData.ToString()=="K")  
                {  
      
                    Cursor.Position = new Point(X, Y);  
                    return true;  
                }  
                return base.ProcessCmdKey(ref msg, keyData);  
            }  
            private void Form1_Load(object sender, EventArgs e)  
            {  
                Bitmap bitmap1 = new Bitmap(this.Width, this.Height);  
                this.DrawToBitmap(bitmap1, this.ClientRectangle);  
                bitmap1 = Example.ConvertToFormat((System.Drawing.Image)bitmap1, PixelFormat.Format24bppRgb);  
                System.Drawing.Bitmap template = Example.ConvertToFormat(System.Drawing.Image.FromFile("1.png"), PixelFormat.Format24bppRgb);  
                // create template matching algorithm's instance  
                // (set similarity threshold to 90%)  
                ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.9f);  
                // find all matchings with specified above similarity  
      
                TemplateMatch[] matchings = tm.ProcessImage(bitmap1, template);  
                // highlight found matchings  
      
                BitmapData data = bitmap1.LockBits(  
                     new Rectangle(0, 0, bitmap1.Width, bitmap1.Height),  
                     ImageLockMode.ReadWrite, bitmap1.PixelFormat);  
                foreach (TemplateMatch m in matchings)  
                {  
      
                    Drawing.Rectangle(data, m.Rectangle, Color.White);  
                    Console.WriteLine(m.Rectangle.Location.ToString());  
                    // do something else with matching  
                    X = m.Rectangle.X;  
                    Y = m.Rectangle.Y;  
                }  
            }  
      
     public static class Example  
        {  
            public static Bitmap ConvertToFormat(this System.Drawing.Image image, PixelFormat format)  
            {  
                Bitmap copy = new Bitmap(image.Width, image.Height, format);  
                using (Graphics gr = Graphics.FromImage(copy))  
                {  
                    gr.DrawImage(image, new Rectangle(0, 0, copy.Width, copy.Height));  
                }  
                return copy;  
            }  
        }  
    

    If you press K, the mouse cursor will move to button1's place.

    Note: It is necessary for you to Maximize Window otherwise the posistion is not correct.

    Result:
    187454-1.gif

    Best Regards,
    Jack


    If the answer is the right solution, please click "Accept Answer" and upvote it.If you have extra questions about this answer, please click "Comment".

    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    0 comments No comments

  2. Valleze 1 Reputation point
    2022-03-29T22:34:31.897+00:00

    I got this code from internet:

    Imports System.Drawing.Imaging
    Imports System.Runtime.InteropServices
    
    Public Class Form1
        Private ImgToFind As New Bitmap("C:\TestFolder\MyIconScreenShot.png")
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
            Dim rect As Rectangle = FindImageOnScreen(ImgToFind, False)
            If rect <> Rectangle.Empty Then
                Dim cntr As Point = New Point(rect.X + CInt(rect.Width / 2), rect.Y + CInt(rect.Height / 2))
                Cursor.Position = cntr
            Else
                MessageBox.Show("Image not found")
            End If
        End Sub
    
        ''' <summary>Finds a matching image on the screen.</summary>
        ''' <param name="bmpMatch">The image to find on the screen.</param>
        ''' <param name="ExactMatch">True finds an exact match (slowerer on large images). False finds a close match (faster on large images).</param>
        ''' <returns>Returns a Rectangle of the found image in sceen coordinates.</returns>
        Private Function FindImageOnScreen(ByVal bmpMatch As Bitmap, ByVal ExactMatch As Boolean) As Rectangle
            Dim ScreenBmp As New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)
            Using g As Graphics = Graphics.FromImage(ScreenBmp)
                g.CopyFromScreen(Screen.PrimaryScreen.Bounds.Location, Point.Empty, Screen.PrimaryScreen.Bounds.Size, CopyPixelOperation.SourceCopy)
            End Using
    
            Dim ImgBmd As BitmapData = bmpMatch.LockBits(New Rectangle(0, 0, bmpMatch.Width, bmpMatch.Height), ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb)
            Dim ScreenBmd As BitmapData = ScreenBmp.LockBits(Screen.PrimaryScreen.Bounds, ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb)
    
            Dim ImgByts((Math.Abs(ImgBmd.Stride) * bmpMatch.Height) - 1) As Byte
            Dim ScreenByts((Math.Abs(ScreenBmd.Stride) * ScreenBmp.Height) - 1) As Byte
    
            Marshal.Copy(ImgBmd.Scan0, ImgByts, 0, ImgByts.Length)
            Marshal.Copy(ScreenBmd.Scan0, ScreenByts, 0, ScreenByts.Length)
    
            Dim FoundMatch As Boolean = False
            Dim rct As Rectangle = Rectangle.Empty
            Dim sindx, iindx As Integer
            Dim spc, ipc As Integer
    
            Dim skpx As Integer = CInt((bmpMatch.Width - 1) / 10)
            If skpx < 1 Or ExactMatch Then skpx = 1
            Dim skpy As Integer = CInt((bmpMatch.Height - 1) / 10)
            If skpy < 1 Or ExactMatch Then skpy = 1
    
            For si As Integer = 0 To ScreenByts.Length - 1 Step 3
                FoundMatch = True
                For iy As Integer = 0 To ImgBmd.Height - 1 Step skpy
                    For ix As Integer = 0 To ImgBmd.Width - 1 Step skpx
                        sindx = (iy * ScreenBmd.Stride) + (ix * 3) + si
                        iindx = (iy * ImgBmd.Stride) + (ix * 3)
                        spc = Color.FromArgb(ScreenByts(sindx + 2), ScreenByts(sindx + 1), ScreenByts(sindx)).ToArgb
                        ipc = Color.FromArgb(ImgByts(iindx + 2), ImgByts(iindx + 1), ImgByts(iindx)).ToArgb
                        If spc <> ipc Then
                            FoundMatch = False
                            iy = ImgBmd.Height - 1
                            ix = ImgBmd.Width - 1
                        End If
                    Next
                Next
                If FoundMatch Then
                    Dim r As Double = si / (ScreenBmp.Width * 3)
                    Dim c As Double = ScreenBmp.Width * (r Mod 1)
                    If r Mod 1 >= 0.5 Then r -= 1
                    rct.X = CInt(c)
                    rct.Y = CInt(r)
                    rct.Width = bmpMatch.Width
                    rct.Height = bmpMatch.Height
                    Exit For
                End If
            Next
            bmpMatch.UnlockBits(ImgBmd)
            ScreenBmp.UnlockBits(ScreenBmd)
            ScreenBmp.Dispose()
            Return rct
        End Function
    End Class
    

    I created a new form and just added a button to test this code above.

    I added to the project:
    AForge code library;
    Aforge.Imaging library;
    AForge.Math library;

    There are many errors in this code, I couldn't fix it, but in this link below, we can see it working in the test that the author of the code published. Apparently, his code works without needing to maximize the form window.

    Could you help me to fix the errors in this code, so that I can try to adapt it to my project? I'm using visual studio 2022, I don't know if this makes a difference to how the code works.

    Here is the link where this code was posted, as an answer to another old question from someone else:

    move-mouse-cursor-to-specific-icon-based-on-saved-image


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.