Remote rendering collisions Unity Hololens

Tomasz Woźniak 0

Hi,
Please tell me - if it is possible to get point of collision Remote Rendered model with Gameobject on Unity scene?
How to get the exact point(s) where the 'local' object collides with the 'remote' model?

Similar question for Hololens - how to get the exact point(s) where a 'remote' object collides with a 'local' spatial environment like floor, walls, etc?

In the image example: the blue table is colliding with the umbrella table. Is it possible to get from ARR point of this collision (where one leg of the blue table is colliding with the round table?)

User's image

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Tomasz Woźniak 0 Reputation points

2023-01-20T20:26:02.73+00:00

Thank you for the answer.
Could you please give some sample or link to the APIs from SDK to detect collisions?

I have found only Spatial Queries (https://learn.microsoft.com/en-us/azure/remote-rendering/overview/features/spatial-queries#spatial-queries-1:~:text=face%20directions%20first.-,Spatial%20queries,-A%20spatial%20query).
But not sure how to use that.
Or maybe is there some other API?
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

2 answers

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
1 deleted comment

Comments have been turned off. Learn more
Jan Krassnigg 91 Reputation points

2023-01-23T11:08:55.65+00:00

Azure Remote Rendering does not have any physics simulation API, what it does have are the spatial queries, as you already discovered: https://learn.microsoft.com/en-us/azure/remote-rendering/overview/features/spatial-queries

The image that you linked is not available anymore, could you please upload that again, so that I can see your use case?

Basically what the spatial queries allow you to do, is to shoot rays into the remotely rendered scene and see where they hit the remote object. With a bit of creativity this functionality allows you to do all sorts of things. You can also ask ARR which objects overlap with a given box or sphere, however, that doesn't give you an exact intersection point.

Say you want to place an object onto a table, you would retrieve the bounds of your object (https://learn.microsoft.com/en-us/azure/remote-rendering/concepts/object-bounds) and then use the lower center position of that to shoot a ray downwards to see where it hits the table. Then you can move the object that distance downwards to have it rest on the table.

However, this only works, if the object that you shoot your ray against is also a remotely rendered object, because only then can the ray see and hit that object.

If you try to work with real-life geometry (such as an actual table that you want to place virtual objects on), you need other methods. In this case you would need to utilize the HoloLens "spatial mapping" functionality (https://learn.microsoft.com/en-us/windows/mixed-reality/design/spatial-mapping). This lets you ask the HoloLens for a mesh representation of the scanned environment. Once you have that mesh, you can theoretically do raycasts against that mesh as well (though ARR would not be involved here).

Here is an article about using spatial mapping with Unity: https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/spatial-mapping-in-unity?tabs=mrtk

I guess that Unity would let you do raycasts againt that mesh, but I'm no expert there.
Please sign in to rate this answer.
Tomasz Woźniak 0 Reputation points

2023-01-23T13:57:39.8433333+00:00

Hi Jan,
Thank you very, very much for this answer.
I put here a lost image.
So my case is to check if it is possible to get point(s) of collision:

a) Remote Rendered object - let's say it is an umbrella with its attached table - with the 'local' gameobject (the one placed in Unity Scene without ARR - blue table)

b) two Remote Rendered objects - let's say the umbrella and blue table are both ARR

c) Remote Rendered object (umbrella) with real-life geometry (mesh generated with spatial mapping) - i.e. to check if the umbrella will pass through the door, or fit in the room, etc.
I already tried Remote Raycast and it works fine.
I was hoping, that spatial queries let us to make query like: get all points of collision between our local mesh and ARR mesh.
As I understand it will give only results for queries like: get all points of collision between ARR mesh and cuboid box.

Or maybe there is some query, where I can pass as parameter of any mesh (not only box)?

If there is no possibility to get points of collision, maybe there is check for cases above, like: is there any collision between those objects?

Thank you in advance for the answers :)

Jan Krassnigg 91 Reputation points

2023-01-23T17:11:29.4333333+00:00

Unfortunately what you are asking for is a very complicated problem in general and there is no easy solution.

Neither for a), b) nor c) there is a built-in solution that would just work, simply because checking these kinds of overlaps between objects is mathematically challenging and physics engines (the code that's used for simulating physical object interactions in games and such) don't handle these kind of situtations, at all (they do things differently to avoid having to deal with such situations in the first place).

So that's why, if you were to do this only locally, using only Unity physics, you also wouldn't find an out-of-the-box easy solution and would still need to find a more creative solution.

Generally, what you can do both in ARR and also in Unity physics, is "overlap queries". Here you ask the system "does this shape overlap with anything in the scene". In ARR we call these operations "spatial queries". It is common that you can only use very simple shapes for such queries: boxes, spheres and sometimes custom convex shapes (ARR does not support the latter).

Additionally you can do raycasts. Admittedly, this is a very limited set of tools, but if you get a bit creative, many problems can be solved.

The main question is exactly what you intend to achieve. Do you want to use this for placement, so that something properly snaps to the ground? Do you want to know whether something would fit somewhere without intersecting real world objects? In case of the latter, is it sufficient to know THAT something intersects, or do you really need to know WHERE it intersects? And how precisely do you need to know the location of the intersection? Would it be sufficient to know that there is an intersection "somewhere in that area" (10cm radius or so)?

Another question is how much control you have over the meshes. Can you preprocess them? I'd guess so, since you need to convert the assets for ARR anyway.

Note that if you want to do spatial meshes from the HoloLens, you can't dynamically send them to ARR, so you would need a local solution to deal with them either way.

Anyway, a couple of ideas:

You could 'voxelize' one of the meshes. E.g. build a box or sphere approximation of it. So for example you would have 10 small spheres for each leg of the table, and a hundred spheres for the tabletop or so. If you have such a representation, you can ask ARR (or Unity) whether any of those spheres intersects with the rest of the scene. You only get a yes/no answer, but since each sphere represents a small area, you can thus pinpoint where the objects intersect. Note that such a voxelization would also be a "convex decomposition", because it represents your object with lots of convex objects (boxes or spheres). See https://github.com/kmammou/v-hacd for reference (though this generates custom convex meshes, which you won't be able to use with ARR).

Another idea would be to shoot rays downwards. Say in your example you take the bounding box of the blue table, and from its top you shoot rays downwards to the bottom of the table. Assuming the blue table is local (so not seen by ARR) the rays can't hit the blue table, only the white one. If any ray hits something within the bounds of the blue table, you have a potential overlap. You could cross check, by also shooting rays locally (against the blue table) and see if anything hits there. So for example at the rounded corner of the table, you may not hit anything in the blue table, so that area is actually empty, and thus any hits in that area in the remote model can be ignored.

As I said, this topic is very complicated and depending on your use case you can find an easy, good-enough heuristic, or you may need to do very complicated things to find a more precise answer.

Let me know whether that was helpful.

Tomasz Woźniak 0 Reputation points

2023-01-23T19:19:18.67+00:00

Yes, this is very, very helpful - thank you very much.

I would like just explain purpose of my questions more precisely.

I am making basic research for a client, to check if there is a possibility to detect a collision between ARR and local, between two ARRs, and between ARR and environment (on HL2 device).

I don't know currently what will be final feature to use this collision detection in ARR, just checking capabilities.

Your answers clarify a lot.

I assume from these answers, that there is no possibility to detect any collision during movement (i.e. when we are grabbing ARR model and placing it in another destination and there is some kind of obstacle). Am I right? There is no Physics Simulation API for Remote Rendering as Santhi suggested?

Once again - thank you for your time :)

Jan Krassnigg 91 Reputation points

2023-01-23T19:44:14.0033333+00:00

Nothing like that is built-in, but there are certainly ways to achieve this. You could use Unity's physics engine to do such a collision simulation on the device. Obviously you need a highly optimized mesh for the physics simulation, but that's actually very common for everything that needs to run in real-time. There are definitely ways to achieve this, but it has to be tailored for the specific use case, there is no one-size-fits-all solution that ARR could provide for you.

Please disregard Santhi's comments, I think this account is a bot that just tries to generate credibility. There is no physics simulation API in ARR.

Tomasz Woźniak 0 Reputation points

2023-01-24T18:29:52.39+00:00

Hi again,

I have, probably the last question.

Is there a way to use SpatialQuerry (not Raycast), to check if there is ARR model inside volume, but not this ARR's bounding box, but mesh?

I mean: SpatialQuery returns Entity - is there a way to check or read from the result if bounds (RemoteRendering.Bounds) passed in this Spatial Query are 'touching' mesh of this ARR model?

Jan Krassnigg 91 Reputation points

2023-01-25T08:35:46.6033333+00:00

No, the spatial queries actually just check whether the query shape overlaps with the bounding box of the remote object. It doesn't check for triangle overlap. I might have implied otherwise in the text above, but looked it up now - it only checks whether the bounds overlap. For everything more detailed, you have to use raycasts.

Tomasz Woźniak 0 Reputation points

2023-01-26T17:28:17.6033333+00:00

Thank you.
All your advices were good - I have solution where I am raycasting from AAR Bounding Box walls position to the opposite walls. This way I have hitpoints on the AAR mesh and then I am checking collisions locally.
That works great.
However, it is not too fast - 'scanning' the whole model takes a lot of time -if the environment (spatial map) changes during scanning - collisions are not correct.

So question is: Is there maybe some Azure service, that could take responsibility for computing collisions in Unity?
We need some kind of solution to make this 'scanning' faster, or something else to compute AAR collisions with spatial environment in realtime like.

Jan Krassnigg 91 Reputation points

2023-01-27T11:54:56.8933333+00:00

I am not aware of any such service.

Generally for ARR a rule of thumb is, that you can do about 10 raycasts every frame without noticing a performance impact. Of course it heavily depends on the actual mesh, but it's a good reference.

Obviously that's not a lot for such cases. Therefore you can't simply scan all the points all the time. You need a more sophisticated approach to determine the result.

Say you divide a 2D area into 20x20 points, giving you 400 sample positions. At 10 rays/frame that would be 40 frames and at 30-60 Hz that would take about a second to get a result for all of them.

I would always shoot the rays from the latest position, so if you have already shot 100 rays and then the user drags the table a bit further, I would shoot the next 300 rays from the latest position.

In theory that could already be a solution. With one second lag you would have a result for all points. But I think we can do much better.

What you are really after is the information that ANYTHING overlaps. So once any ray reports a hit, you may not need to sample all the others anymore.

I would also not sample the points in a stritcly increasing order. Meaning if I sampled the left corner of the table in the last frame, then I don't need to sample that corner + 1 centimeter in this frame. It makes more sense to rather sample the right corner of the table instead. You could either just randomly pick positions to sample from, or have some elaborate scheme, either would increase your chances to find the blocking hit earlier.

Finally, once you did find a hit, you need to validate that it continues to be a blocking hit over time. I would record the indices for sample points where hits were found, and recheck them every frame. So at maximum you would have 10 rays that are blocked, that you recheck every frame. As long as any of them hit, the complete object is blocked. If any of those rays become unblocked, I can remove it from my list and instead sample another random position again.

There are probably many other ways how you can implement such a stochastic sampling approach, I just wanted to give you an idea. Generally I don't think you need a perfect and instant solution, you just need one that leads to results reasonably fast and "eventually correct" :-)
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Remote rendering collisions Unity Hololens

2 answers

Your answer