September 2016
Volume 31 Number 9
[Unity]
Building Virtual Reality Applications
By Tim Kulp
Virtual reality (VR) provides a unique opportunity and challenge for application developers. On one hand, there’s opportunity to build immersive experiences unbound by the existing 2D UX paradigm. On the other hand, there’s the challenge to build a great VR experience that’s more than a 3D view of a 2D screen. Fortunately, some fundamentals of app development and UI design are similar, such as creating a master/details view, getting input from the user, and communicating across applications via services. In this article, I’ll explore these topics from the context of building a business application in VR so that you can see how to start your own VR apps.
Why VR?
It’s a good time to enter VR app development with the availability of great tools such as Unity and the relatively small VR marketplace. Think about the current mobile app market space. There are millions of apps and new apps have a lot of competition. The VR market space is still small, however, making it easier to stand out and get noticed. On top of the market size, there’s a great hunger currently for VR content with early adopters. How to build apps in this space is still a new challenge, with best practices evolving. Early adopters in VR development will help shape the future of this platform. It’s an exciting time to be in VR.
Building apps for VR devices such as Samsung Gear VR and Google Cardboard is a great entry point for developers to start building their content for low-cost distribution channels. If built with a mind toward Mixed Reality, VR apps can be ported to the HoloLens because the tools are the same and the content can be the same. The only difference is the code to perform tasks (such as Gaze vs. Raycast, which I will explore later in this article), and utilizing some of the HoloLens unique features such as speech, spatial mapping and gesture. With a little research and experimentation, you can convert your VR app to a HoloLens app.
I won’t go into exhaustive detail on setting up a VR project. Instead, I recommend exploring the Unity VR Samples project in the Unity Asset Store (bit.ly/1qYLBX9). The project built in this article will start from scratch, but will use scripts from the Unity VR Samples project to leverage existing code for standard tasks (such as making an object interactive). I recommend creating a blank Unity project and importing the Unity VR Samples for reference outside of the project that’ll be built with this article. Unity VR Samples has a lot of great content but is built to be a standalone app and requires some extra configuration for running a scene outside the included scenes. Keep Unity VR Samples open and available because it has a few scripts that will be used in this project. For this article, I’ll use Unity 5.3 with the Android tools enabled so that I can build the app for the Samsung Gear VR device.
Contoso Travel
Contoso Travel is a corporate travel system that lets a travel administrator see the bookings for their team members. Travel administrators can view travelers by airport, search for a specific city, and view a traveler’s trip details. In the app, I want to create a rich experience that shows the spatial relationship of travelers for Contoso. Normally this would be a line-of-business app, mobile or Web, delivered to the users through a browser. For this article, I’ll build this app in VR to give users a deep experience, use avatars to represent the travelers and show where people are in relation to each other within 3D space.
VR opens many possibilities for an app such as this. From the travel administrator’s perspective, VR can be a more personal experience, engaging the traveler’s avatars instead of just displaying dots on a map or a name on a list. For travelers, VR can give travelers a preview of the office they’re visiting or sites to see while in a remote location. The idea here is to create an immersive experience that users can enjoy.
Starting Out
You’ll want to prepare this project as a mobile VR experience. To start, create a new 3D Unity Project called ContosoTravelVR. First, change the timing of the world to a slower framerate. In the Edit | Project Settings | Time inspector, set Fixed Timestep = 0.01666666, set Maximum Allowed Timestep = 0.1666666 and Timescale = 1. Maximum Allowed Timestep slows down Unity’s physics time so that physics calculations do not impact framerate. While physics time and framerate are not directly related, the CPU power required for physics calculation can create issues with processing frames. Setting Maximum Allowed Timestep puts a maximum limit on how much time to spend on physics updates. Remember that in a mobile VR world, the device is drawing two images every frame: one for the left eye and one for the right eye. Complex computations processing at higher frame rates can cause a device, such as a phone, to get frames out of sync, causing each eye to see something different. That causes disorientation and nausea in users. Obviously, this is critical to avoid for a great user experience. Avoid anything that could have a performance impact, such as high polygon counts for objects, long-running scripts or locking the thread.
Next, update the Player Settings for mobile VR. To do this, go to Edit | Project Settings | Player and select the Other Settings section. Within this section there’s a setting called Virtual Reality Supported. Check the box to ensure that the world is recognized as a VR app. This setting will automatically convert the Main Camera object that already exists in Unity to a VR camera.
Now that the app is set, set up the scene. In the Hierarchy window, add a cube with the scale of x=10, y=0.1 and z=10. Reset the position of the cube to be x=0, y=0 and z=0 to center the flattened cube in the world. This will provide a floor for your world, which gives users a sense of space and grounding, as well as a place for the entities in the world on which to stand. Just like in 2D apps, you need to give the user a sense of where the content and interactions of the app will occur. This helps users orient themselves in the app. For ContosoTravel, all actions will occur within the bounds of this floor object.
In this world, you need an object that will marshal data throughout the app. Create an Empty Game Object in the Hierarchy window and rename the object to TravelerManager. This object will be the source of all the travel data within the app. While selecting the TravelerManager game object, click Add Component | New Script. Name the script TravelerManager and click Create and Add. This will create a new script to handle the logic of the TravelerManager.
As a travel app, you need to see travelers in the world. For this, add a new Game Object to the Hierarchy and call it TravelerTemplate. This will be the object that’s shown for each traveler represented in the system. If you’re familiar with prefabs, use a character you’ve designed or something from the Asset Store. For my app, I’ll use some low-poly characters I found in the Asset Store that’ll give the world a fun, blocky look. Any game object will work for the TravelerTemplate—a prefab, a cube, a sphere and so on. Once the TravelerTemplate is added, disable the game object by clicking on the checkbox next to the game object’s name in the Inspector window. This will make the TravelerTemplate disappear from the screen.
Finally, add the following scripts from the Unity VR Samples project to your project: VREyeRaycast, VRInput, VRInteractiveItem and Reticle. I’ll discuss the exact purpose of these scripts as I use them throughout the article, but they’ll save me a lot of coding later in the app.
Setting Up the TravelerManager Class
In Unity, double-click the TravelerManager script to open it in MonoDevelop or Visual Studio. Add the code in Figure 1 to the component.
Figure 1 TravelerManager Script Component
public GameObject TravelerTemplate;
public List<Traveler> TravelerList;
void Awake () {
TravelerList = new List<Traveler>();
}
public void LoadTravelers() {
try
{
// Add GetTravelers here
}
catch(Exception ex)
{
Debug.Log(ex.ToString());
}
}
The TravelerTemplate will hold the Prefab I’m using to create the travelers on the screen. Remember, TravelerTemplate can be any game object: a cube, a sphere, a prefab character and so on. Also, by marking this variable public, the value can be set in the Unity design tools, which let the design team control the traveler template without needing to re-code the TravelerManager. You also create two lists, one for storing all the travelers created as GameObjects, and the other to maintain the data for each traveler. Next, you’ll add the input mechanisms to load the travelers and give users something to do in the application.
Hooking Up a Button
With the basic world set up, you need to collect input from the user to get the app doing stuff. To start, add a button that will trigger the LoadTravelers method of the TravelerManager script component. Why not just load the travelers by default? In this case, you want to provide a point of orientation for the user to understand what’s happening in the app. VR is too new for users to have predefined expectations of what’s going to happen as soon as they load a world. UI instructions are a good tool to help the user orient and acclimate to the new world.
Create a new Empty Game Object on the root of the application and call it GUI. This game object will hold the UI components for the app. With the GUI game object selected, add a new Canvas object. The Canvas object holds all UI components on the screen. Once the Canvas object is created, go to the Inspector and change the Render Mode to World Space. This will detach the canvas from the camera and let content be placed in the world, as opposed to stuck to the camera. Now add a new Button object to the Canvas. Set the button’s text to “Load Travelers.” Change the button’s display transform to x=1.87, y=.05, z=0 and width = 9; height = 3 with a scale of x=0.25, y=0.25, which makes the button smaller and in the view of the user.
Select the button object in the Hierarchy window and then select the plus sign on the On Click list (in the Inspector window) to add a new click event. Buttons can have as many click events as needed, which lets you use a single button to trigger many different events. This is good for triggering an action and then disabling or hiding the button. Each click event requires a game object and a function available to that game object to be defined. After adding a click event, drag the TravelerManager game object to the Object field for the click event. Once the TravelerManager is set, the Function list will show the LoadTravelers method. If the function list doesn’t show LoadTravelers, you need to build your code project in Visual Studio and try again to select LoadTravelers. Select the plus sign on the click event list again to add another event. Drag the button object from the Hierarchy to the Object field for the new click event, select GameObject.isActive, and leave the checkbox empty. This means that when the button is clicked, Unity will first execute LoadTravelers and then set the isActive value of the button to false to hide the button. From a user perspective, this button provides initial instructions. Once those instructions are complete, the instructions are no longer needed so you remove them.
Getting Data from a Service
Modern apps get their data from many different places. In this app, traveler data is provided by the travel.contoso.com API. Getting data from a URL is very simple in Unity, letting apps build out data-rich experiences. At the bottom of the TravelerManager script, add a new class called Traveler. This will be your data object from the traveler service:
public class Traveler{
public int Id { get; set; }
public string Name { get; set; }
public string Title { get; set; }
public string DepartureCity { get; set; }
public string ArrivalCity { get; set; }
}
Now, you need to get data to fill this class. Create a new method under LoadTravelers called GetTravelers. This method will connect to the API endpoint, download the JSON object provided by the API and then send the result to another method to show the data request results on screen:
private IEnumerator GetTravelers(){
var www = new WWW("https://travel.contoso.com/api/Traveler");
yield return www;
if (www.isDone)
DisplayTravelers(www.text);
}
Now replace the “add GetTravelers here” comment in LoadTravelers with the following line of code:
StartCoroutine(GetTravelers());
LoadTravelers will call GetTravelers, which pulls the data from the API. The data from the API is sent to the DisplayTravelers method for further processing. To start this, use StartCoroutine, which executes GetTravelers without locking the thread and essentially freezing the app until execution is complete. Creating this smooth flow of code is one of the steps you take to ensure that the app doesn’t drop frames. Coroutines start and pause a method with each frame, letting a method span multiple frames without locking the app. StartCoroutine is very useful to do methods that could be long-running, such as waiting for data to return from a Web service. Coroutines can also be used for methods on large numbers of objects. For example, if the traveler service returned hundreds of people traveling—coroutines could be useful for calling actions across all the people, perhaps to emulate the appearance of wandering for the travelers. For more information on coroutines, see the Unity documentation at bit.ly/2ahxSBu.
In GetTravelers the WWW object from Unity is used to connect to the provided URL. Using the yield keyword, the WWW object is returned until it’s complete by checking the isDone property. Once the download from the URL is complete, isDone will equal true, which will then pass the text returned by the API call to the DisplayTravelers method. The WWW object can be used to post data, as well as get data, so that the app could have functionality to add travelers in the future. Full documentation for the WWW object on the Unity site is at bit.ly/2alFoML.
Then you take the result from the API and convert it to something the user will be able to see in the app. Before you can show travelers in different locations on the map, you need to create those locations in the world. Go back to the Unity designer and create an empty Game Object on the root of the Hierarchy tree. Call this empty game object Location Container. Set Location Container to be at position X=0, Y=0, Z=0 (center of the world). With the Location Container selected, create a few empty Game Object children scattered around the floor cube created when the article started. These empty GameObjects represent airports; the floor is like a map. In the DisplayTravelers method, you look at the traveler’s destination city and match it to one of these empty Game Objects. Rename the empty Game Objects to some airport names. I used BWI, MSP and SFO. To complete the setup of airports, you create a tag for the Game Objects called Airport so that you can find all the airport items by searching for a single tag value. Creating a tag for a Game Object is done by going to the top left of the Inspector for the object, clicking on the tag dropdown list and selecting New Tag. You can create the new tag while all empty Game Objects are selected, which lets you set the tag for all the objects at once.
Add the code shown in Figure 2 after the getTravelers method.
Figure 2 Convert JSON Traveler Objects to Unity Game Objects
private void DisplayTravelers(string json){
var jsonTravelers = new JSONObject(json);
foreach (var jsonTraveler in jsonTravelers.list){
// Convert jsonobject to traveler object
Traveler traveler = ConvertJSON(jsonTraveler);
GameObject travelerRep =
Instantiate(TravelerTemplate,
new Vector3(0f, 0.05f, 0f),
TravelerTemplate.transform.rotation) as GameObject;
travelerRep.name = traveler.Name;
GameObject[] airports = GameObject.FindGameObjectsWithTag("Airport");
foreach (var airport in airports) {
if (airport.name == traveler.ArrivalCity) {
travelerRep.transform.position =
new Vector3(airport.transform.position.x, 0.05f,
airport.transform.position.z);
break;
}
}
TravelerList.Add(traveler);
}
}
Using the free JSONObject script from the Unity Asset store (bit.ly/2akN9p9), you’re able to convert the JSON text to a Traveler C# code object. From there, using the Instantiate method, you create a copy of the TravelerTemplate, set it to the default coordinates, and name the Game Object the name of the traveler. This name is important for later in the app as it will be used to identify the traveler from the traveler list. Next, you find all the Game Objects tagged as Airports. This provides a collection of Airport objects that you can go through and find the matching object by name to place the traveler at the corresponding airport location. Using the tags you set up earlier lets you easily find the Airport Game Objects. At the end of the method, you add the traveler object to the TravelerList so that you can reference the traveler data later in the app. Now you have a master list of travelers displayed in your world. It’s time to let users select a traveler so that you can show the detail information for that traveler.
Selecting a Traveler
Thinking about the world that’s been created so far, a great way to do a details view is to have a speech bubble come from the avatar like the traveler is saying where they’re traveling to. This would be a fun way to show the details while providing a UI in close proximity to the selected traveler.
In VR, selecting a traveler is done by looking at the object. To know if the user is looking at a traveler you need to use the Raycast object. Raycast is an invisible beam that emanates from the camera and lets the system know when an object with a collider component on it intersects with the beam. Using a Raycast lets your code detect when the user is looking at an object that should provide an action. Earlier you imported the VREyeRaycast script, which implements all the logic necessary to create a Raycast on a camera. Add the VREyeRaycast script to the Main Camera. You’ll also need to add the Reticle script to the Main Camera to provide a UI component to show where the user is looking.
With Raycast setup on the camera, you need to update the TravelerTemplate to know how to react to an intersection with the Raycast. To do this, add the VRInteractiveItem script to the TravelerTemplate. This script is a quick setup for working with events like when the Raycast is over an object, or when the Raycast leaves an object, or even capturing the click event.
Next, you create a script called TravelerInteraction, which implements the events in the VRInteractiveItem script. For this app, you’ll use the OnOver, OnOut and OnClick events. When the user looks at a Traveler object, the Traveler will change in appearance. When the user looks away, the Traveler object will change back. When the user is looking at the Traveler object and clicks, a dialog will appear showing the traveler’s name, title and destination. Here’s the TravelerInteraction script:
public class TravelerInteraction : MonoBehavior{
public VRInteractiveItem InteractiveItem;
public Shader Highlight;
public TravelerManager TravelerManager;
public Canvas Dialog;
private Shader standardShader;
private bool isDialogShowing;
}
The VRInteractiveItem is needed to work with the VREyeRaycast and assign methods to the VRInteractiveItem events. Highlight is a custom shader that’ll be used to highlight the Traveler object when the user looks at the object. I won’t go into custom shaders here as they’re a complex topic. To learn more, see the documentation on the Unity site at bit.ly/2agfb7q. TravelerManager is a reference to the TravelerManager script, which will be used to look up details about the selected Traveler from the TravelerManager.TravelerList collection. The final public property here is the Canvas game object that will be used to show details about the traveler when clicked. Private variables include a reference to the standard shader to return the Traveler object to after you stop looking at the traveler and a Boolean value to determine if the dialog is showing.
To get started, add the Enabled method to the TravelerInteraction class to connect the events of the VRInteractiveItem class to methods within the TravelerInteraction class:
private void OnEnable(){
InteractiveItem.OnOver += HandleOver;
InteractiveItem.OnOut += HandleOut;
InteractiveItem.OnClick += HandleClick;
}
HandleOver uses the Renderer component to change the shader to a custom shader. The Highlight property lets a designer use the Unity Editor to set the Shader for highlighting the Traveler.
private void HandleOver(){
var renderer =
gameObject.GetComponentInChildren<Renderer>();
renderer.material.shader = Highlight;
}
private void HandleOut(){
var renderer =
gameObject.GetComponentInChildren<Renderer>();
renderer.material.shader = standardShader;
}
When the user looks at a Traveler, the Highlight shader is applied to differentiate the Traveler from the other travelers. This provides a visual queue to the user and whichever Traveler currently has the user’s gaze. When the user looks away the HandleOut method will be triggered and return the Traveler to the standardShader value.
Now, you want to show the traveler information to users when they click on the traveler. To do this, first you need to add System.Linq to the using statements for the TravelerInteractive class so that you can find the traveler by the game object’s name. Also, you need to add the UnityEngine.UI namespace to work with the Text component in the dialog, as shown in Figure 3.
Figure 3 Display Dialog When Object Is Clicked
private void HandleClick(){
if (!isDialogShowing){
var textComponent =
Dialog.gameObject
.GetComponentInChildren<Text>();
var travelerInfo =
TravelerManager.TravelerList.SingleOrDefault(x => x.Name ==
this.gameObject.name);
textComponent.text = String.Format(“{0}\n{1}”,
travelerInfo.Name, travelerInfo.DestCity);
Dialog.gameObject.SetActive(true);
isDialogShowing = true;}
else{
Dialog.gameObject.SetActive(false);
isDialogShowing = false;}
}
In this code block, first you determine if the dialog is displayed or not. If it is displayed, then disable the dialog to hide it from view. If the dialog isn’t showing, then get the text object and set its value to the name and destination of the Travelers. When you originally created the traveler object, you used the traveler name as the name of the game object used to represent the traveler. Using LINQ, you can search the traveler list by that name to return the traveler’s detail information. For simplicity sake you use the traveler’s name but in larger, more complex systems, a unique identifier would be most ideal to use so that each traveler is unique. A quick note on LINQ, remember that performance is key in building a mobile VR experience. LINQ can do great things—but it does have overhead—so be aware of where LINQ is used and determine if it’s needed or just a convenience. Here LINQ is used for brevity. Once the text is set to the name and destination city of the traveler, the dialog is enabled (which is a canvas object) to display the text to the user.
Wrapping Up
In this article, I reviewed the basics of building a UI, getting data from a service and interacting with objects in a VR world. Using these skills, the world of ContosoTravel can grow into a rich app with search, perhaps more levels to represent regions of the world and animations to bring the avatars to life (such as wandering around). Explore these fundamentals to build VR worlds for today and prepare for the mixed reality users of tomorrow.
Tim Kulp is the principal engineer at bwell in Baltimore, Md. He’s a Web, mobile and UWP app developer, as well as author, painter, dad and “wannabe Mad Scientist Maker.” Find him on Twitter: @seccode or via LinkedIn: linkedin.com/in/timkulp.
Thanks to the following Microsoft technical expert for reviewing this article: Adam Tuliper