Design and develop in mixed reality


When it comes to mixed reality projects, familiar design processes that you or your team uses don't automatically transfer to designing 3D content. For instance, when you're brainstorming a registration screen for a mobile app or website, a simple rectangle is enough.

Designing for three dimensions instead of two can be more time consuming. Drawing or sketching in 3D is much harder than in 2D. Some 3D tools exist for gaming and engineering, but they usually require training. This lack of fast, lightweight tools can affect your design pipeline and overload the person who's developing the application.

Luckily, a few techniques can help effectively iterate through ideas and get your content fleshed out. These techniques also allow different team members and disciplines to have a voice.

Diagram of a design process with steps for bodystorming, acting, and storyboarding.


Bodystorming is the process of prototyping interactions, digital objects, UI, and animations by using crafting materials. Anyone involved in the design process can try out the proposed interactions, features, and mechanics without having to fully prototype them. The goal of all bodystorming sessions is to get actionable input, get testing feedback, and think spatially about the app content.

Photograph of a design team prototyping mixed reality interactions and features with crafting materials.

Acting and feedback

After you've been through a bodystorming session or two, the next step is to stage how a user would move through your mixed reality experience. The acting phase is all about testing ideas in three-dimensional space, but also about allowing others to watch the interactions and events from the outside and give feedback. This approach is especially important when you have nontechnical team members or stakeholders. For example, if you're exploring a mixed reality experience that's designed for hospitals, acting out your thinking to a medical professional can provide invaluable feedback.

Photograph of a design team using crafting materials to act out and receive feedback on mixed reality content and interactions.


Choosing the best method for getting your ideas and proposed experience always depends on the target audience. If you need to present some new ideas to a development team, bodystorming and acting out the experience or new content can be more than enough to get the job done. But when you need to convince stakeholders or nontechnical team members, you might need to opt in for more high-fidelity assets like storyboarding.

Diagram of a high-fidelity storyboard example.

Whether the storyboards are detailed or just sketches meant to evoke the broad strokes of an experience is up to you.

Mixed reality development tools

When it comes to working with holograms in your applications, it's easiest to use some of Microsoft's premade development tools, like the Mixed Reality Toolkit (MRTK) for Unity and Unreal Engine. MRTK is an open-source, cross-platform development kit that provides an input system, foundational components, and common building blocks for spatial interactions.