How does the MAUI app about Android Channel implement automatic clicking at any coordinate of a given screen

Question

How does the MAUI app about Android Channel implement automatic clicking at any coordinate of a given screen

liu 0

I am facing such a need, I want to use the MAUI app to operate my phone.

Of course, this process involves clicking, swiping, and finding the right place to operate.

My intention is to use opencv to locate the specified operation position by means of picture comparison.

However, for me, there are some problems that cannot be ignored at this time.

How do I configure Android permissions on MAUI
How to click the specified Android screen position

These two questions puzzled me for a few days, these days I read a lot of documents, blogs. I didn't get a proper answer.

All I got was a few fragments of code, and I even lost a lot of hair.

So I wanted to get help in the community to get a solution.

Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-21T06:44:35.8866667+00:00

Hello,

My intention is to use opencv to locate the specified operation position by means of picture comparison.

opencv is a third-party and it's not supported on Q&A. You could visit their GitHub Repo and try binding a Java Library for MAUI or connect the owners to support MAUI. (This doc is about Xamarin, please adjust a MAUI version)

How do I configure Android permissions on MAUI

Please see Permissions - .NET MAUI | Microsoft Learn. If you want to use opencv, you could view their doc and check which permissions are required, then set the matching attributes in the Android Manifest file.

How to click the specified Android screen position

Please try Android.View.MotionEvent. For more details, please see Touch in Android - Xamarin | Microsoft Learn (This doc is about Xamarin.Android, it also applies to .net-android and MAU)

Best Regards,

Wenyan Zhang

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
liu 0 Reputation points

2023-08-21T09:14:15.96+00:00

I appreciate your reply.

About configuring Android permissions on MAUI

I checked the Micosoft document again according to what you said

But nothing else has changed, and I don't find the permissions I need there

Instead, an extension was found underneath. Is that where it happens on its own?

About automatically clicking on the specified Android screen location

I also looked through the touch section of the Xamarin documentation, but probably because I'm not

I am very familiar with Xamarin, so the documentation is not very friendly to me.

I don't really understand the gestures in Xamarin with my need to "define automatic click location"

What's the direct connection, I think it's more like a monitor on the screen......

When the action occurs.......

Do you have a better solution for the above problem of MAUI?
liu 0 Reputation points

2023-08-21T09:31:17.3066667+00:00

Here are some of the Microsoft documents I looked through.
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-21T09:48:25.26+00:00

For example: if you want to add the Camera permission, you could open the AndroidMainfest.xml file, then drop down to Required permissions section, and select CAMERA. (Or open the AndroidMainfest.xml file with Source code, then add <uses-permission android:name="android.permission.CAMERA" />)
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-21T09:57:43.2666667+00:00

As I understood, you need a touch event. So, it's recommended that you try Android.View.MotionEvent. Looks like this is not what you want. Could you describe more about how you want to "define automatic click location"? Is this a clicking method? And I'm not sure if you can achieve this function by opencv. Do you want to get the location via opencv, and compare with some other control's position?
liu 0 Reputation points

2023-08-22T01:08:14.24+00:00
Yes, I need to manually add permissions in "AndroidManifest.xml" myself.

For example, as you said, adding a camera permission requires' < uses-permission android:name="android.permission.CAMERA" /> 'code snippet

However, I did not find a snippet in the Microsoft documentation requesting 'permission code for Android to automatically click'
liu 0 Reputation points

2023-08-22T01:09:21.3833333+00:00
I introduced opencvsharp implementation in winform to find the coordinate position of a specified element from an image.

I want to do the same in MAUI, I first place the element images that I need to locate in the resource folder, and when I need to use them, I take instant screenshots,

Then take the element image you need to find from the resource file and compare the two. And that gives me the coordinates of where I need to click in the screenshot.

In this process, the entire Android screen may have corresponding coordinates, I do not understand 'Android.view.motionevent'

How do custom gestures help me in a necessary way
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-22T09:30:14.96+00:00

I'm not sure what "permission code for Android to automatically click" means. Users can allow or disallow permissions, and developers cannot let applications automatically allow permissions.

Do you mean request the Camera permission? If so, you could refer to the example, and replace the Permissions.LocationWhenInUse with Permissions.Camera .

For the second question, you want to calculate the specific position, is there a problem similar to this one?
liu 0 Reputation points

2023-08-23T08:06:38.6366667+00:00

"Permission code for Android to automatically click" means:

I need to tap the screen once, and the location is my choice. Something like,

Use adb shell instruction under Android

"adb shell input tap x y"

x and y are coordinates.

The permission of this thing. How to implement in C#?

For example, in Java, as long as you get the root permission, you can use Runtime.getRuntime to execute shell instructions on Android.

This translates to a problem that needs to be solved:

I need to develop Android applications in MAUI that operate automatically under certain conditions

My cell phone.

In C#, how do I apply root permission to Android to see. How do I send it to an Android phone

shell instruction.

I am not aware of the code support for their implementation, such as you say

"Permissions. LocationWhenInUsePermissions. Camera" to add a permission

I did not find the code definition for root permissions in the Microsoft MAUI permissions
liu 0 Reputation points

2023-08-23T08:08:22.61+00:00

"Permission code for Android to automatically click" means:

I need to tap the screen once, and the location is my choice. Something like,

Use adb shell instruction under Android

"adb shell input tap x y"

x and y are coordinates.

The permission of this thing. How to implement in C#?

For example, in Java, as long as you get the root permission, you can use Runtime.getRuntime to execute shell instructions on Android.

This translates to a problem that needs to be solved:

I need to develop Android applications in MAUI that operate automatically under certain conditions

My cell phone.

In C#, how do I apply root permission to Android to see. How do I send it to an Android phone

shell instruction.

I am not aware of the code support for their implementation, such as you say

"Permissions. LocationWhenInUsePermissions. Camera" to add a permission

I did not find the code definition for root permissions in the Microsoft MAUI permissions
liu 0 Reputation points

2023-08-24T00:58:09.79+00:00

Here is the logic code I implemented.
liu 0 Reputation points

2023-08-24T01:03:23.1733333+00:00

In this method, I start a process that accesses "/system/xbin/su" to get ROOT permission. Finally, a bool value is returned to determine whether the permission was successfully obtained.However, when the code path executes the Start0 method, an error is printed
liu 0 Reputation points

2023-08-24T01:05:31.26+00:00

The error is as follows, and it tells me that I do not have permission to access this path
liu 0 Reputation points

2023-08-24T01:07:43.0366667+00:00

This is my "AndroidManifest.xml" configurati,How should I face this problem?
Anonymous

2023-08-24T08:45:42.2666667+00:00

I start a process that accesses "/system/xbin/su" to get ROOT permission.

Above way actually result that processes can be run as root, not your MAUI applications themselves.

The error is as follows, and it tells me that I do not have permission to access this path

Without root permission, your application cannot access it.
liu 0 Reputation points

2023-08-25T01:18:45.7+00:00

Yes "The above approach actually causes the process to run as root,

not your MAUI application itself."I can't agree more with what you're saying. My code essentially causes the process to run as root,

rather than giving permission directly to my MAUI application itselfYes,

the machine I used for testing was not ROOT, and I don't have a ROOT machine on hand.

This left me without a perfect test environment, and I was hesitant to call the system's identity directly in this way for a machine that had already been ROOT,

run ROOT as the process, and then execute the "adb shell" command.
Anonymous

2023-08-29T09:15:36.5766667+00:00

the machine I used for testing was not ROOT, and I don't have a ROOT machine on hand.

You can get root permission easily in android emulator for testing, firstly, open your android emulator, then open adb command(Open a command prompt and go to the folder where adb is installed. If the Android SDK is installed at its default location, adb is located at C:\Program Files (x86)\Android\android-sdk\platform-tools\adb.exe; if not, modify this path for the location of the Android SDK on your computer.), input adb root command, then you can get root permission for your android emulator.

Your answer

Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-21T06:44:35.8866667+00:00

Hello,

My intention is to use opencv to locate the specified operation position by means of picture comparison.

opencv is a third-party and it's not supported on Q&A. You could visit their GitHub Repo and try binding a Java Library for MAUI or connect the owners to support MAUI. (This doc is about Xamarin, please adjust a MAUI version)

How do I configure Android permissions on MAUI

Please see Permissions - .NET MAUI | Microsoft Learn. If you want to use opencv, you could view their doc and check which permissions are required, then set the matching attributes in the Android Manifest file.

How to click the specified Android screen position

Please try Android.View.MotionEvent. For more details, please see Touch in Android - Xamarin | Microsoft Learn (This doc is about Xamarin.Android, it also applies to .net-android and MAU)

Best Regards,

Wenyan Zhang

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
liu 0 Reputation points

2023-08-21T09:14:15.96+00:00

I appreciate your reply.

About configuring Android permissions on MAUI

I checked the Micosoft document again according to what you said

But nothing else has changed, and I don't find the permissions I need there

Instead, an extension was found underneath. Is that where it happens on its own?

About automatically clicking on the specified Android screen location

I also looked through the touch section of the Xamarin documentation, but probably because I'm not

I am very familiar with Xamarin, so the documentation is not very friendly to me.

I don't really understand the gestures in Xamarin with my need to "define automatic click location"

What's the direct connection, I think it's more like a monitor on the screen......

When the action occurs.......

Do you have a better solution for the above problem of MAUI?
liu 0 Reputation points

2023-08-21T09:31:17.3066667+00:00

Here are some of the Microsoft documents I looked through.
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-21T09:48:25.26+00:00

For example: if you want to add the Camera permission, you could open the AndroidMainfest.xml file, then drop down to Required permissions section, and select CAMERA. (Or open the AndroidMainfest.xml file with Source code, then add <uses-permission android:name="android.permission.CAMERA" />)
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-21T09:57:43.2666667+00:00

As I understood, you need a touch event. So, it's recommended that you try Android.View.MotionEvent. Looks like this is not what you want. Could you describe more about how you want to "define automatic click location"? Is this a clicking method? And I'm not sure if you can achieve this function by opencv. Do you want to get the location via opencv, and compare with some other control's position?
liu 0 Reputation points

2023-08-22T01:08:14.24+00:00

Yes, I need to manually add permissions in "AndroidManifest.xml" myself.

For example, as you said, adding a camera permission requires' < uses-permission android:name="android.permission.CAMERA" /> 'code snippet

However, I did not find a snippet in the Microsoft documentation requesting 'permission code for Android to automatically click'
liu 0 Reputation points

2023-08-22T01:09:21.3833333+00:00

I introduced opencvsharp implementation in winform to find the coordinate position of a specified element from an image.

I want to do the same in MAUI, I first place the element images that I need to locate in the resource folder, and when I need to use them, I take instant screenshots,

Then take the element image you need to find from the resource file and compare the two. And that gives me the coordinates of where I need to click in the screenshot.

In this process, the entire Android screen may have corresponding coordinates, I do not understand 'Android.view.motionevent'

How do custom gestures help me in a necessary way
Wenyan Zhang (Shanghai Wicresoft Co,.Ltd.) 36,436 Reputation points Microsoft External Staff

2023-08-22T09:30:14.96+00:00

I'm not sure what "permission code for Android to automatically click" means. Users can allow or disallow permissions, and developers cannot let applications automatically allow permissions.

Do you mean request the Camera permission? If so, you could refer to the example, and replace the Permissions.LocationWhenInUse with Permissions.Camera .

For the second question, you want to calculate the specific position, is there a problem similar to this one?
liu 0 Reputation points

2023-08-23T08:06:38.6366667+00:00

"Permission code for Android to automatically click" means:

I need to tap the screen once, and the location is my choice. Something like,

Use adb shell instruction under Android

"adb shell input tap x y"

x and y are coordinates.

The permission of this thing. How to implement in C#?

For example, in Java, as long as you get the root permission, you can use Runtime.getRuntime to execute shell instructions on Android.

This translates to a problem that needs to be solved:

I need to develop Android applications in MAUI that operate automatically under certain conditions

My cell phone.

In C#, how do I apply root permission to Android to see. How do I send it to an Android phone

shell instruction.

I am not aware of the code support for their implementation, such as you say

"Permissions. LocationWhenInUsePermissions. Camera" to add a permission

I did not find the code definition for root permissions in the Microsoft MAUI permissions
liu 0 Reputation points

2023-08-23T08:08:22.61+00:00

"Permission code for Android to automatically click" means:

I need to tap the screen once, and the location is my choice. Something like,

Use adb shell instruction under Android

"adb shell input tap x y"

x and y are coordinates.

The permission of this thing. How to implement in C#?

For example, in Java, as long as you get the root permission, you can use Runtime.getRuntime to execute shell instructions on Android.

This translates to a problem that needs to be solved:

I need to develop Android applications in MAUI that operate automatically under certain conditions

My cell phone.

In C#, how do I apply root permission to Android to see. How do I send it to an Android phone

shell instruction.

I am not aware of the code support for their implementation, such as you say

"Permissions. LocationWhenInUsePermissions. Camera" to add a permission

I did not find the code definition for root permissions in the Microsoft MAUI permissions
liu 0 Reputation points

2023-08-24T00:58:09.79+00:00

Here is the logic code I implemented.
liu 0 Reputation points

2023-08-24T01:03:23.1733333+00:00

In this method, I start a process that accesses "/system/xbin/su" to get ROOT permission. Finally, a bool value is returned to determine whether the permission was successfully obtained.However, when the code path executes the Start0 method, an error is printed
liu 0 Reputation points

2023-08-24T01:05:31.26+00:00

The error is as follows, and it tells me that I do not have permission to access this path
liu 0 Reputation points

2023-08-24T01:07:43.0366667+00:00

This is my "AndroidManifest.xml" configurati,How should I face this problem?
Anonymous

2023-08-24T08:45:42.2666667+00:00

I start a process that accesses "/system/xbin/su" to get ROOT permission.

Above way actually result that processes can be run as root, not your MAUI applications themselves.

The error is as follows, and it tells me that I do not have permission to access this path

Without root permission, your application cannot access it.
liu 0 Reputation points

2023-08-25T01:18:45.7+00:00

Yes "The above approach actually causes the process to run as root,

not your MAUI application itself."I can't agree more with what you're saying. My code essentially causes the process to run as root,

rather than giving permission directly to my MAUI application itselfYes,

the machine I used for testing was not ROOT, and I don't have a ROOT machine on hand.

This left me without a perfect test environment, and I was hesitant to call the system's identity directly in this way for a machine that had already been ROOT,

run ROOT as the process, and then execute the "adb shell" command.
Anonymous

2023-08-29T09:15:36.5766667+00:00

the machine I used for testing was not ROOT, and I don't have a ROOT machine on hand.

You can get root permission easily in android emulator for testing, firstly, open your android emulator, then open adb command(Open a command prompt and go to the folder where adb is installed. If the Android SDK is installed at its default location, adb is located at C:\Program Files (x86)\Android\android-sdk\platform-tools\adb.exe; if not, modify this path for the location of the Android SDK on your computer.), input adb root command, then you can get root permission for your android emulator.

Share via

How does the MAUI app about Android Channel implement automatic clicking at any coordinate of a given screen

Your answer