Silly Computer, Do What I Mean!

Using node-tail to Stream VSTS Log Output

j.seghers — Wed, 01 Aug 2018 14:22:51 GMT

I have not done much with Typescript, JavaScript, node.js, etc. I've been much more involved in the compiler-driven, strong type checking, C++/C# and even Java worlds. Nonetheless, some environments dictate the terms of engagement. I've been setting up Continuous Integration for a set of projects written in C++ and C# for Windows and Unity 3D. The code is stored in GIT on Visual Studio Team Services (VSTS, visualstudio.com) and I am running the builds using VSTS build agents. I set up an Azure VM to act as a custom build agent, onto which I've installed a separately-licensed installation of Unity and the necessary other compilers and SDKs. Running a Unity batch build in that environment can be handled in various ways. You can create an MsBuild task to execute Unity. You can have a batch file to run it. Or you can create a custom VSTS build task written in either PowerShell or node.js. I chose to do the latter and wrote the task using Typescript. When a build doesn't work, it's really helpful to have the full build log available. Most compilers send this information to stdout and the normal VSTS build results page will stream this in near real time to your web browser as well as hold onto it in the build results. Unity doesn't do this, however. Unity sends this entire output to the logfile for the session which is normally %LOCALAPPDATA%\Unity\Editor\Editor.log. This is not an appropriate location on a build agent where you want to keep all of your file references within your agent's working directory for the current build. You can direct Unity where to place the logfile with the Unity command line argument -logfile . So my first iteration of my Unity build task used the vsts-task-lib to start the Unity batch build, wait for the build to complete, and then used cmd.exe to type the logfile. This succeeded in placing the build output in the overall build logs, but I found the delay with no output to be annoying. Unity builds can run for tens of minutes even for relatively small applications, so having some way of gauging progress is nice.

Tail to the Rescue

The Unix command tail is commonly used for monitoring the log files of background tasks. But this presumes you have Unix-like tools and access to run them on the log file in question. I was already using the async/await pattern to wait for Unity to finish processing. I should be able to stream the contents of the log file to the console. I found that there is just such a package available for JavaScript: node-tail. To use it easily with typescript, you'll want to use npm to install both the library itself and the @types package:

npm install tail
npm install @types/tail

Following the usage information in the README of the node-tail package, I used the default settings when constructing the tail object. My code was essentially:

Start Unity running
Wait in a loop until fs.stat(logfile) showed the logfile exists.
Start tail to read the logfile and send the output to the console asynchronously.
Wait for Unity to finish
Wait for tail to finish streaming the logfile.

There were two problems with this: there is no direct support for Step 5 in node-tail; and even after working around that, I would get the very first part of the log, a long delay, and then the rest of the log after Unity finished. Initially I thought this was a problem with the VSTS system that is very much a product with ongoing development. But further experiments showed that was not the case.

Two Ways of Watching a File

Node.js has two fs module apis that generate events when changes are made to a file: fs.watch and fs.watchFile. According to the documentation, fs.watch is more efficient, but not available on all systems. fs.watchFile is based on polling fs.stat. It makes sense that node-tail defaults to using the former. Unfortunately, in the environment in which the build agent is running, this did not detect changes to the log file size until the log file was closed by Unity. However, setting the node-tail option useWatchFile = true allows node-tail to properly detect that the file has changed and stream the new data.

Test for Completion

The public interface for tail consists of construction, the methods watch and unwatch, and events "line" and "error". There is no event for "done" or "eof". This is by design because most uses of tail are for watching open-ended logs. A quick perusal of the JavaScript code in the package shows that tail sets up callbacks for when the log file changes. On each change it records the start and end points of the "new" section of the file and enqueues this to a work queue. The readblock method then processes each queue entry by reading the indicated portion of the file to a buffer and fires "line" events for each line of text. After Unity has finished running, we know the logfile is at its final size. Therefore we can watch the tail.pos value (records the largest size it has seen) to be the same as the log file size and the tail.queue.length to detect when the queue is empty. When both of these conditions are met, we can shut down the tail by calling tail.unwatch(). Unfortunately, tail.pos and tail.queue are not available in the public interface defined in the @types/tail package. To get around this, we use two accessor functions within which the type is defined as "any" which allows arbitrary (non-type-safe) access to members:

function getTailPos(t:any) : number {
    return t.pos;
}
function getTailQueueLength(t:any) : number {
    return t.queue.length;
}

The resultant code looks something like this (I've removed some error- and edge case-handling):

    var unity = buildUnityCommand();
    var execResult = unity.exec();

    // wait for log file to be created
    while (execResult.isPending && !fs.existsSync(logfile)) {
        await sleep(1000);
    }

    console.log("========= UNITY BUILD LOG ==========")

    var logTail = new tail.Tail(logfile, { fromBeginning: true, follow: true,
                logger: console, useWatchFile: true,
                fsWatchOptions: { interval: 1009 } });

    logTail.on("line", function(data) { console.log(data); });
    logTail.on("error", function(error) { console.log('ERROR: ', error); });

    var result = await execResult;
    var size = fs.statSync(logfile).size;

    while (size > getTailPos(logTail) || getTailQueueLength(logTail) > 0) {
        await sleep(2089);
    }

    logTail.unwatch();

    console.log("======== UNITY BUILD LOG END ========");

Acknowledgements

I'd like to thank Luca Grulla, the author of node-tail, for making this library available.

Creative Prototyping in Unity

j.seghers — Fri, 02 Dec 2016 12:46:47 GMT

My colleague Alex Bennett and I gave a talk at the Unite 2016 conference in Los Angeles about how we use Unity as a creative prototyping and aggregation tool. Unity has posted 78 videos from the conference on a YouTube list. Here is the link to our talk Creative Prototyping in Unity

Multitouch Event System for Unity 5.5 (and 5.3, 5.4)

j.seghers — Tue, 15 Nov 2016 13:29:13 GMT

I mentioned in my first post about Unity Multitouch processing that the 5.5 beta is making some private items protected to enable better derivation. This enables integration of the Multitouch Event System possible without having to build it into UnityEngine.UI. There is another way of doing such integration--via reflection. This gets rather cumbersome and the compiler cannot help you get things right. As it turns out, the changes to allow integration without reflection did not get made until Beta 11 (and even at this point there's still one that isn't available yet). So the new repository, https://bitbucket.org/johnseghersmsft/unity-multitouch, may change if this need for reflection changes, but it's a kind of sneaky back door to get the multi-selection working, and Unity may choose to do that differently :) This repository has five branches: 5.3, 5.4, 5.5, 5.3.testonly, and 5.3.testonly. The first three branches provide a Multitouch test application along with the Multitouch Event System in the Plugins directory. The *.testonly branches only provide the test application for working with the UnityEngine.UI fork referenced in my prior blog entry. Branch 5.5 has two tags on it for Beta 10 and Beta 11. The Beta 10 version has to use the same reflection methods as 5.3 and 5.4. At this point, the new repository should be considered the most up-to-date project and the custom UnityEngine.UI version will not be updated going forward.

The Test App

[caption id="attachment_115" align="alignright" width="300"] Screenshot of multitouch test app (click to expand)[/caption] The test app has a single scene, "main". It has a Scroll View on the left with a bunch of simple items in it. There is a normal button. And there is a target panel with a script which handles multi-touch events. On the right side of the screen is also a text control showing debugging information about the touch points and how they are being processed. Each of the controls has a script which collects information about received events (TouchStatusCollector) so they can be displayed by this debugging tool. Since the editor does not support multitouch, it is best to build the app and run it outside of the editor to get the full sense of how it works. You can get some sense of it in the editor (or on non-touch screens) by using a keyboard cheat: hold down the Control key when clicking with the mouse. This will cause a "fake touch" to be held onto as though it is still active which will let you simulate pinch and rotation gestures, but not the two-fingered Pan gesture since that requires both points to be moving in the same direction. Along with these controls, there is an object called TouchTrackerOverlay. This contains a script that displays the touch points and the clusters with which they are associated. For example, this image shows how three touches in a cluster might look, with the green lines illustrating the spanning tree which connects the points. [caption id="attachment_125" align="alignright" width="223"] Screengrab showing a three-touch cluster as rendered by the ShowTouchSpanningTree debugging component.[/caption] Play with this a little. Use both hands and interact with multiple controls at once. Try starting with your hands close together and them pull them apart and you will see one cluster split into two. Move them back together and the clusters will stay separated. Each time you add a new touch, it is added to the cluster closest to the touch--if it's close enough. The maximum distance is set on the Multiselect Event System component of the EventSystem game object. I've made it short (3 inches) for this demo. In normal usage, on a large display, something in the 5-6 inch range may be more appropriate.

Unity Mouse Events

The normal Unity EventSystem is designed much more around mouse clicks than around touch. When the user presses the left mouse button while pointing at a button on the screen, for example, Unity Event System performs a raycast from that point. The raycast identifies all of the objects which are shown (or would be shown) at that pixel on the screen. In the case of Button1, there are three such objects: the Canvas, the Button1 game object, and the Text game object. Canvas rendering places child objects on top of parent objects, so the Text object is the top-most game object at that pixel. The EventSystem then executes the OnPointerDown event on the hierarchy starting by the topmost object. So it examines the Text object of Button1 to see if it has a handler for the event (i.e. if there is a component that implements the IPointerDownHandler interface). The Text object does not, so it walks up the tree to the parent: Button1. The Button component does implement IPointerDownHandler--so it becomes the target object for handling the further mouse events until the mouse button is released. This can be seen by examining the PointerEventData object for future mouse events, where the eventData.pointerPress member is set to this Button's game object.

Two Touches, One Event

When you touch the screen with one finger, Unity essentially does the same thing as above. However, when you touch with two fingers, Unity's normal processing doesn't handle things the way you might want. It still generates a single OnPointerDown event, and other pointer events such as drag and release. However, it uses the centroid of the touches as the screen point for the raycast used to determine the event target. So if you touch the screen with two fingers on either side of a button, the centroid (the common center point of all of the touches) will be halfway between your fingers, thus sending the events to the button. Similarly, if you're running on an 84" Surface Hub, and two people press the screen on opposite sides, something in the middle will be pressed. The object handling the events could examine the Input.touches array to look at all of the individual touch points.

Touch Clusters

To solve this I group touches into clusters, and each cluster is processed similarly to the way a single mouse click is handled. In each cluster, the earliest touch (or the first one processed in the case of simultaneous touches) is considered the Active Touch and is used for defining the raycast position. In the Surface Hub case above, there would be two events, one for each cluster on either side of the display. Each event contains information about the points in the associated cluster. (Note that a cluster may only have a single touch.)

Handling Multitouch Events

Let's take a look at the script attached to the multitouch target panel: Assets/Samples/TouchSizePosition.cs.

[RequireComponent(typeof(RectTransform))] 
public class TouchSizePosition : UIBehaviour, 
    IPanHandler, IPinchHandler, IRotateHandler, 
    IPointerDownHandler, IPointerClickHandler

The RequireComponent annotation tells us that the Transform for the game object on which the component is placed must be a RectTransform. The interfaces provide the ability to receive Pan (the multifinger equivalent of the Unity-defined Drag), Pinch, and Rotate gestures as well as the normal pointer click handlers.

public void OnPan(SimpleGestures sender, MultiTouchPointerEventData eventData, Vector2 delta)
{
    var rcTransform = GetTransformAndSetLastSibling();
    TranslateByScreenDelta(rcTransform, eventData.delta, eventData.pressEventCamera);
    eventData.eligibleForClick = false;
}
private RectTransform GetTransformAndSetLastSibling()
{
    var rcTransform = (RectTransform)transform;
    rcTransform.SetAsLastSibling();
    return rcTransform;
}
public static void TranslateByScreenDelta(RectTransform rc, Vector2 delta, Camera cam)
{
    var pos = rc.anchoredPosition;
    var parent = (RectTransform)rc.parent;
    var worldPos = parent.TransformPoint(pos);
    var screenPoint = RectTransformUtility.WorldToScreenPoint(cam, worldPos);
    screenPoint += delta;
    RectTransformUtility.ScreenPointToLocalPointInRectangle(parent, screenPoint, cam, out pos);
    rc.anchoredPosition = pos;
}

IPanHandler defines a single method, OnPan. The event is sent by the SimpleGestures Multitouch Gesture Module. (In a future blog post I'll dive more into Gesture Modules--the code that interprets finger motions.) The eventData parameter provides all the information normally available in a PointerEventData (MultiToucPointerEventData derives from the former) as well as the information about the touch cluster and certain information about the change in the touch cluster--such as the position of the cluster's centroid. In the case of multiple such targets, I want the one currently being moved to also be the top-most visual object, thus the helper function GetTransformAndSetLastSibling is used in the handlers. TranslateByScreenDelta uses the event parameter delta to compute how much to move the RectTransform's anchoredPosition. (Note that the delta parameter is the same as the eventData.centroidDelta value.)

public void OnPinch(SimpleGestures sender, MultiTouchPointerEventData eventData, 
    Vector2 pinchDelta)
{
    var rcTransform = GetTransformAndSetLastSibling();
    var size = rcTransform.rect.size;
    float ratio = Mathf.Sign(pinchDelta.x) * (pinchDelta.magnitude / size.magnitude);
    size *= 1.0f + ratio * PinchFactor;
    size = Vector2.Max(size, MinimumSize);
    size = Vector2.Min(size, MaximumSize);
    rcTransform.SetSizeWithCurrentAnchors(RectTransform.Axis.Horizontal, size.x);
    rcTransform.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, size.y);
}

The IPinchHandler also defines a single method, OnPinch. Here the pinchDelta is the change in distance between the points forming the pinch. This is pretty straightforward for two fingers, but it works for more fingers as well. If you're curious, this is an aggregate of how much each touch moves toward or away from the centroid.

public void OnRotate(SimpleGestures sender, MultiTouchPointerEventData eventData, float delta)
{
    var rcTransform = GetTransformAndSetLastSibling();
    rcTransform.Rotate(new Vector3(0, 0, -delta));
}

Similarly, the OnRotate method is the implementation of the IRotateHandler interface. The delta passed here is an angle (in degrees) based on the change in angle of each point from the centroid.

Normal Unity Events

The normal Unity pointer events work as expected--but there are a few things to note. As I mentioned earlier, many multitouch gestures begin as a single touch since we humans are not necessarily going to touch the screen at exactly the same moment with each finger. Therefore, the IPointerDownHandler will be invoked prior to many multitouch events. OnPointerDown is often not used to perform an action, but may change the visual state of a control. The Button, for example, will change color to show it is pressed (purple in the sample application). The other reason to implement IPointerDownHandler is to inform the Unity Event system that this control wants to receive the OnPointerUp and OnPointerClick events. If some control ControlA only implements IPointerClickHandler, and no other component in the parent hierarchy implements IPointerDownHandler, then ControlA will receive the OnPointerClick event. However, if ControlP is the parent (or further ancestor in the hierarchy) of ControlA implements IPointerDownHandler, then ControlP will be considered the target for all of the future such events, and ControlA will not receive the OnPointerClick event.

public void OnPointerDown(PointerEventData eventData)
{
    eventData.Use();
}

public void OnPointerClick(PointerEventData eventData)
{
    var rcTransform = GetTransformAndSetLastSibling();
    rcTransform.SetSizeWithCurrentAnchors(RectTransform.Axis.Horizontal, _originalSize.x);
    rcTransform.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, _originalSize.y);
    rcTransform.rotation = _originalRotation;
}

Disambiguation

Try this with the built test application: scroll the list up and down with one finger, then add a second finger so you're dragging with two fingers. See how the list stops scrolling as soon as the second finger is added? And it continues to ignore the drag if you remove the second finger. When the SimpleGestures module identifies a cluster for which it needs to send an event, it also clears any single-press state for that cluster. Therefore, the ScrollRect sees an OnEndDrag event and an OnPointerUp event (but no OnPointerClick). At this point, single-touch processing is disabled for this cluster. There are also two selections on the SimpleGesture module which affects how other Gesture Modules are affected by either enough touch points being seen in a cluster and/or by events being sent. These control whether other Gesture Modules are allowed to process the current input. More on this in a later post.

Displaying Events for Debugging

You will note that the debugging information shows the Scroll Rect receiving OnDrag events when only one finger is used, and OnPan events after a second finger is added. This debugging information is being collected by adding the TouchStatusCollector script to any game object for which you want this information captured. This component automatically registers itself with the ShowTouchStatus component to contribute debugging information to that display. One thing to note about Unity Event processing: When a game object receives an event, ALL components on that game object will receive the event if they have implemented the interfaces. So adding TouchStatusCollector to a control that is already receiving those events will not affect them. However, since TouchStatusCollector implements most of the interfaces, it may affect the handling of those events by parents of the object.

Using the Multitouch Event System

[caption id="attachment_55" align="alignright" width="200"] EventSystem game object showing components for Multitouch processing (click to expand)[/caption] Copy the Assets/Plugins/Multitouch directory from this test app into the same location in your project. Then add an EventSystem object (or edit the one you already have--Unity will automatically add one when you add a Canvas object if there is no EventSystem in your scene) with the components shown here (MultiselectEventSystem, Win10TouchInputModule, SimpleGestures, and MouseTouches). Then implement the interfaces for the Multitouch events you wish to handle on the components you wish to handle them.

Multitouch Event System for Unity 5.4.x

j.seghers — Fri, 04 Nov 2016 15:23:35 GMT

NOTE: I now have a version for Unity 5.3 and 5.4 which use reflection to integrate with UnityEngine.UI instead of having to build a custom verison. Please see my Unity 5.5 post for this new version.

In my previous blog entry, I mentioned that the repository at https://bitbucket.org/johnseghersmsft/ui had the version of my Multitouch event system for Unity 5.3.x. I have now updated the repo with the 5.4 branch merged in. So simply choose the branch that matches the version you are using. The 5.4 branch has been tested with Unity 5.4.1p2.

Multitouch Event System for Unity

j.seghers — Mon, 31 Oct 2016 12:08:19 GMT

I get to play with some cool toys on the Envisioning team such as Surface Hub and other large display devices that can track up to 100 touch points simultaneously. These devices encourage multiple people to interact at one time. Unity, unfortunately, doesn't handle this very well. While the Input.touches array will show the touches, Unity's event handling for things like selected object, button clicks, drags, etc., treats the centroid of the points as the action point. If you have two people trying to interact on opposite sides of the screen, they are treated as one person interacting somewhere in the middle. I need to work with gestures from one to five fingers at a time, in multiple clusters. Most of the multitouch gesture systems available for Unity do not integrate with the Unity event system. As such, it can be quite difficult to determine whether a Unity uGUI control (like a ScrollRect or Button) should handle the input, or some other element which is responding to a gesture recognizer. So I forked the Unity.UI project to solve these issues. My modifications include:

MultiselectEventSystem: this component extends EventSystem to enable the new functionality
Win10TouchInputModule: this component extends StandaloneInputModule to process the touch points into clusters.
SimpleInputGestures: This component plugs into the gesture recognition pipeline to fire UnityEvents for Pan, Pinch, and Rotate gestures. The gesture pipeline is extensible by adding similar components to the EventSystem GameObject.

NOTE: I now have a version for Unity 5.3 and 5.4 which use reflection to integrate with UnityEngine.UI instead of having to build a custom verison. Please see my Unity 5.5 post for this new version.

Using the Multitouch Event System is accomplished by including these components on a GameObject in your scene. (Click to Enlarge) Then you can use the IPanHandler, IPinchHandler, and IRotateHandler interfaces to receive these multitouch events. These events carry a MultiTouchPointerEventData object which extends PointerEventData. This contains information about the cluster of touch points that caused the event to fire. Things like the centroid position and delta and the TouchCluster describing the points. Here's an example of processing the OnPinch event which applies a pinch event to the local scale of an object called HandRoot:

public void OnPinch(SimpleGestures sender, MultiTouchPointerEventData eventData, Vector2 pinchDelta)
{
    float ratio = pinchDelta.magnitude / displaySize.magnitude;
    ratio *= Mathf.Sign(pinchDelta.x);
    ratio = 1.0f + (ratio * PinchFactor);

    var scale = HandRoot.transform.localScale * ratio;
    HandRoot.transform.localScale = scale;
}

One piece of magic that MultitouchEventSystem performs is to track multiple selected objects. During the processing for each cluster, the currentSelectedGameObject represents the selected object for that cluster. This allows normal Unity uGUI controls to treat the clusters (even if each is a single touch point) as if they were the only touches happening. Thus if you have two or more people interacting on the same screen, the normal controls will react to each of them simultaneously. The current state of my fork, https://bitbucket.org/johnseghersmsft/ui, targets Unity 5.3.x. I'm in the process of updating it to handle 5.4.x (relatively minor changes). The later versions of 5.5 beta include changes which allow the proper derivation of these classes without having to build a custom UnityEngine.UI subsystem. There were a number of methods and fields which needed to be protected instead of private. I will be updating my additions and publishing a new repository with only these files so that they can be used with the default 5.5 subsystems. I will be demonstrating this system briefly as part of my talk at Unite 2016 Los Angeles, 10am November 3rd.

Who I Am and What I Am Writing About

j.seghers — Thu, 20 Oct 2016 13:00:34 GMT

I am currently a developer on the Office Experience Organization (OXO) Envisioning team. We are a design-led group (which means we developers are quite outnumbered) who explore future experiences and trends in the productivity experience in a 3-7 year time window. Ours is an exploratory and aspirational view rather than a prescriptive one. One of the vehicles we use to communicate our explorations is the production of videos. Examples of these are at the following links:

We also tell stories driven by demonstrations in Microsoft's Envisioning Center where we use custom crafted installations to bring to life slices of our visions. In doing this we often push the edges of the tools we use. One of those tools is the Unity 3D development system. Unity is a great system for creating games and other interactive environments in both 2D and 3D. The well-worn paths are well documented both in Unity's documentation and in tutorials across the web. The edges, however, can be obscure and the web often has incomplete information about how to do the things we want to make Unity do. In my explorations I seem to push the edges rather frequently. Much of what I will post here are the solutions I have found. I don't claim that they will be complete, but hopefully they will help the web's information about Unity be a little less incomplete.