How does the MAUI app about Android Channel implement automatic clicking at any coordinate of a given screen
I am facing such a need, I want to use the MAUI app to operate my phone.
Of course, this process involves clicking, swiping, and finding the right place to operate.
My intention is to use opencv to locate the specified operation position by means of picture comparison.
However, for me, there are some problems that cannot be ignored at this time.
- How do I configure Android permissions on MAUI
- How to click the specified Android screen position
These two questions puzzled me for a few days, these days I read a lot of documents, blogs. I didn't get a proper answer.
All I got was a few fragments of code, and I even lost a lot of hair.
So I wanted to get help in the community to get a solution.
Developer technologies | .NET | .NET MAUI
-
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) • 36,436 Reputation points • Microsoft External Staff
2023-08-21T06:44:35.8866667+00:00 Hello,
My intention is to use opencv to locate the specified operation position by means of picture comparison.
opencv
is a third-party and it's not supported on Q&A. You could visit their GitHub Repo and try binding a Java Library for MAUI or connect the owners to support MAUI. (This doc is about Xamarin, please adjust a MAUI version)How do I configure Android permissions on MAUI
Please see Permissions - .NET MAUI | Microsoft Learn. If you want to use
opencv
, you could view their doc and check which permissions are required, then set the matching attributes in the Android Manifest file.How to click the specified Android screen position
Please try Android.View.MotionEvent. For more details, please see Touch in Android - Xamarin | Microsoft Learn (This doc is about Xamarin.Android, it also applies to .net-android and MAU)
Best Regards,
Wenyan Zhang
If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
-
liu • 0 Reputation points
2023-08-21T09:14:15.96+00:00 I appreciate your reply.
About configuring Android permissions on MAUI
I checked the Micosoft document again according to what you said
But nothing else has changed, and I don't find the permissions I need there
Instead, an extension was found underneath. Is that where it happens on its own?
About automatically clicking on the specified Android screen location
I also looked through the touch section of the Xamarin documentation, but probably because I'm not
I am very familiar with Xamarin, so the documentation is not very friendly to me.
I don't really understand the gestures in Xamarin with my need to "define automatic click location"
What's the direct connection, I think it's more like a monitor on the screen......
When the action occurs.......
Do you have a better solution for the above problem of MAUI?
-
liu • 0 Reputation points
2023-08-21T09:31:17.3066667+00:00 Here are some of the Microsoft documents I looked through.
-
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) • 36,436 Reputation points • Microsoft External Staff
2023-08-21T09:48:25.26+00:00 For example: if you want to add the
Camera
permission, you could open theAndroidMainfest.xml
file, then drop down to Required permissions section, and selectCAMERA
. (Or open theAndroidMainfest.xml
file with Source code, then add<uses-permission android:name="android.permission.CAMERA" />
) -
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) • 36,436 Reputation points • Microsoft External Staff
2023-08-21T09:57:43.2666667+00:00 As I understood, you need a touch event. So, it's recommended that you try Android.View.MotionEvent. Looks like this is not what you want. Could you describe more about how you want to "define automatic click location"? Is this a clicking method? And I'm not sure if you can achieve this function by
opencv
. Do you want to get the location viaopencv
, and compare with some other control's position? -
liu • 0 Reputation points
2023-08-22T01:08:14.24+00:00 - Yes, I need to manually add permissions in "AndroidManifest.xml" myself.
For example, as you said, adding a camera permission requires' < uses-permission android:name="android.permission.CAMERA" /> 'code snippet
However, I did not find a snippet in the Microsoft documentation requesting 'permission code for Android to automatically click'
-
liu • 0 Reputation points
2023-08-22T01:09:21.3833333+00:00 - I introduced opencvsharp implementation in winform to find the coordinate position of a specified element from an image.
I want to do the same in MAUI, I first place the element images that I need to locate in the resource folder, and when I need to use them, I take instant screenshots,
Then take the element image you need to find from the resource file and compare the two. And that gives me the coordinates of where I need to click in the screenshot.
In this process, the entire Android screen may have corresponding coordinates, I do not understand 'Android.view.motionevent'
How do custom gestures help me in a necessary way
-
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) • 36,436 Reputation points • Microsoft External Staff
2023-08-22T09:30:14.96+00:00 I'm not sure what "permission code for Android to automatically click" means. Users can allow or disallow permissions, and developers cannot let applications automatically allow permissions.
Do you mean request the Camera permission? If so, you could refer to the example, and replace the
Permissions.LocationWhenInUse
withPermissions.Camera
.For the second question, you want to calculate the specific position, is there a problem similar to this one?
-
liu • 0 Reputation points
2023-08-23T08:06:38.6366667+00:00 "Permission code for Android to automatically click" means:
I need to tap the screen once, and the location is my choice. Something like,
Use adb shell instruction under Android
"adb shell input tap x y"
x and y are coordinates.
The permission of this thing. How to implement in C#?
For example, in Java, as long as you get the root permission, you can use Runtime.getRuntime to execute shell instructions on Android.
This translates to a problem that needs to be solved:
I need to develop Android applications in MAUI that operate automatically under certain conditions
My cell phone.
In C#, how do I apply root permission to Android to see. How do I send it to an Android phone
shell instruction.
I am not aware of the code support for their implementation, such as you say
"Permissions. LocationWhenInUsePermissions. Camera" to add a permission
I did not find the code definition for root permissions in the Microsoft MAUI permissions
-
liu • 0 Reputation points
2023-08-23T08:08:22.61+00:00 "Permission code for Android to automatically click" means:
I need to tap the screen once, and the location is my choice. Something like,
Use adb shell instruction under Android
"adb shell input tap x y"
x and y are coordinates.
The permission of this thing. How to implement in C#?
For example, in Java, as long as you get the root permission, you can use Runtime.getRuntime to execute shell instructions on Android.
This translates to a problem that needs to be solved:
I need to develop Android applications in MAUI that operate automatically under certain conditions
My cell phone.
In C#, how do I apply root permission to Android to see. How do I send it to an Android phone
shell instruction.
I am not aware of the code support for their implementation, such as you say
"Permissions. LocationWhenInUsePermissions. Camera" to add a permission
I did not find the code definition for root permissions in the Microsoft MAUI permissions
-
liu • 0 Reputation points
2023-08-24T00:58:09.79+00:00 Here is the logic code I implemented.
-
liu • 0 Reputation points
2023-08-24T01:03:23.1733333+00:00 In this method, I start a process that accesses "/system/xbin/su" to get ROOT permission. Finally, a bool value is returned to determine whether the permission was successfully obtained.However, when the code path executes the Start0 method, an error is printed
-
liu • 0 Reputation points
2023-08-24T01:05:31.26+00:00 The error is as follows, and it tells me that I do not have permission to access this path
-
liu • 0 Reputation points
2023-08-24T01:07:43.0366667+00:00 This is my "AndroidManifest.xml" configurati,How should I face this problem?
-
Anonymous
2023-08-24T08:45:42.2666667+00:00 I start a process that accesses "/system/xbin/su" to get ROOT permission.
Above way actually result that processes can be run as root, not your MAUI applications themselves.
The error is as follows, and it tells me that I do not have permission to access this path
Without root permission, your application cannot access it.
-
liu • 0 Reputation points
2023-08-25T01:18:45.7+00:00 Yes "The above approach actually causes the process to run as root,
not your MAUI application itself."I can't agree more with what you're saying. My code essentially causes the process to run as root,
rather than giving permission directly to my MAUI application itselfYes,
the machine I used for testing was not ROOT, and I don't have a ROOT machine on hand.
This left me without a perfect test environment, and I was hesitant to call the system's identity directly in this way for a machine that had already been ROOT,
run ROOT as the process, and then execute the "adb shell" command.
-
Anonymous
2023-08-29T09:15:36.5766667+00:00 the machine I used for testing was not ROOT, and I don't have a ROOT machine on hand.
You can get root permission easily in android emulator for testing, firstly, open your android emulator, then open adb command(Open a command prompt and go to the folder where adb is installed. If the Android SDK is installed at its default location, adb is located at C:\Program Files (x86)\Android\android-sdk\platform-tools\adb.exe; if not, modify this path for the location of the Android SDK on your computer.), input
adb root
command, then you can get root permission for your android emulator.
Sign in to comment