The Google Tokyo Tree

This was a project from my time at Thinkwell Group. It was the project I worked on that was probably the one that mixed the most interesting technologies: Unity, C#, C++, OpenCV, Arduino, OSC, Beaglebone (not Raspberry Pi), ultrasonic sensor, infrared cameras and infrared light projectors, all to create one seamless experience.

The goal of the task

The given user journey was for a tree that dispensed knowledge. The user was to be able to stand in front of the tree and ask questions and the tree would answer. We knew that guests would come as a group. Immediately, I thought, if the guests would come as a group, it would be a pleasant feeling to simulate an airflow and that Unity, with its physics simulation, should be able to do that. It ended up being one of the defining features of our experience.

The tree went through many variations and a lot of testing. It is, as far as I know, currently it the offices of Google Tokyo. This is a description of what the final version was as late as 2016.

Asking

When the guests approach the tree, it is just a stylized tree, with ambient sound and music (written by Thadeus Reed) playing. The wall is shaped like the 3D image itself. There is a shape projected onto the floor inviting guests to come forward in a nonverbal way. This is an image of the tree with testing lines in the Tokyo building:

Here’s a wider angled view, showing the lobby itself:

When a guests stands the project circle, they are detected by a ultrasonic sensor mounted on an Arduino board. The sensor just communicates how far the object in front of it is. Usually, it’s the floor, but when someone steps on the circle, the distance gets much shorter. The Arduino board is connected to a Beaglebone system-on-chip computer that runs an extremely simple program that just forwards this information via OSC to Unity. When Unity detects that value has gotten beneath a given threshold, it can safely assume that someone is standing on the circle and the shape of the projected circle changes. Then they are explicitly asked on the wall in front of them: “Please tell me what image you want to see”, either in English or in Japanese. This eventually got changed to sequence of more specific questions to keep the guests engaged. There’s also a logo that explicitly tells them that the tree is in audio capture mode or in effect, listening.

Seamless input, to an extent

Then the guest speaks to the tree. That ended up being a lot more trickier than we expected. Among the many unexpected situations were:

How should the system react if the guest steps away before the search is complete? (This happened almost every time, even though we assumed it wouldn’t.)
What should happen if the guest changes their mind in the middle of a query?
What should happen if the guest does not understand what is expected of them?
What should be done if none of the guests step up to the tree? (Should someone from the building invite them to?)

When the guest does make their query, the words are transcribed to text in real time thanks to a Google speech recognition system. We had a hard time testing it in Los Angeles because it was surprisingly difficult to find Japanese speakers. When we did get to Tokyo, the microphone installation that we had used that was chosen to be completely invisible had an echo. I remember it was a big problem, but since I’m not an audio person, I don’t remember all the details.

Here’s a tester making a request to the tree. Notice how the pattern on the floor has changed:

What I do remember is that the guests were split into four categories: male English speaking guests, female English speaking guests, male Japanese speaking guests, and finally female Japanese-speaking guests. It was very difficult for the system to understand the last of those. Women Japanese speakers did not always get their words recognized. Of course, they were category that had been the least tested in Los Angeles.

Answers

Once a request is made, it is sent to the Google Images API via Unity. Some of leaves change their textures to the top ten results. The leaves would always flutter (not really, more about that later) in the airflow caused by the guests walking by them, but from that point on, if the guests walk by them too quickly, the leaves can actually fall down and flutter downwards, still influenced by the airflow of the guests’ motion.

Here is the tree in Tokyo showing flowers:

The original user journey had the guests being able to kick the leaves when they were on the ground. Unfortunately, that was not possible. the problem was that the leaves projected on ground came from projectors on the ceiling and so when guests leaned forward to look at them, the leaves were hidden by the guests shadow. Also, our system was not good at detecting specific motions of depth.

Here’s Ellie from Google demonstrating the query feature.

Airflow

When the leaves have changed to images, the modified leaves respond to people moving in front of the tree as if a virtual airflow were created by their motion.

Here are testers moving by the tree to make the leaves fall:

Here’s Ellie from Google demonstrating the airflow feature.

To do this, the wall is filmed with an infrared camera. The camera has to be infrared, otherwise the projection itself would be filmed, creating an infinite loop. With an infrared camera, we project infrared light and we only see the guests themselves and their motion. We then feed these images to OpenCV and analyze the with the Farneback optical flow algorithm using C++. This produces a vector matrix that tells us exactly how much the content of each image has shifted between two frames, in other words, what the airflow is. This is then sent to Unity using C#’s marshalling abilities, basically, sharing a C array between C# and C++. We then adapt the raw data in the way explained by Ellie above so that the leaves respond to the motion in a more cartoony way. If we just followed real-life physics, the leaves would barely respond to our motion at all.

Flutter

Before falling down, the leaves flutter as the guests move by. I could never get the leaves to flutter correctly. Then, a teammate had a better idea: just move the leaves’ normals around. It worked extremely well. That made the light bounce around the leaves as if they were fluttering without having to find the right motion. The edges didn’t move, but in this case, that was a good thing because they needed to keep their overall assembled shape. It was an elegantly simple solution to what could have been a very messy problem.

3D wall

It shows that the wall the tree was projected was shaped to look like the wall of the projection. This was a nice idea in theory. But in practice, it means that the projection and the physical wall would always shift away from each other after a short while in real life. To keep them lined up, a maintenance system was set up. This is what it looked like:

Conclusion

The project was a creative challenge and very rewarding. It was definitely an unusual use of the Unity game engine and a mix of technologies that don’t come together often. Still, we did manage to make a smooth, seamless experience in the end, that welcomes everyone at the Google Partner Plex Tokyo.

SEECQUEL at the ETC

Back in my Fall 2012 semester at the Entertainment technology Center at Carnegie Mellon University, I took part in a project that we called SEECQUEL, because it was the second part of the SEEC project, SEEC being the Schrader Environmental Education Center at Oglebay, in West Virginia.

The Goal

Our task was to use an electronic device to encourage children to depend less on their electronic devices. It was an essentially paradoxical task. We used a heavily ruggedised Android tablet. It was a military grade tablet that children could drop on any surface without it breaking. It was expensive and we did not actually test that. The software platform we used was the Unity game engine with Scaleform for the interface. Scaleform allows a Flash layer on top of Unity for the interface.

At the end of tho project, these were our testers who were in fact very respectful of all the hardware that was handed to them.

Augmented Reality before its time

Devices like the Hololens did not exist at the time. But, in the context of the period, what we had built really was augmented reality. We added content to a natural space. We did not just add messages and items with content, we actually had a device that filmed the park itself and added visual context superimposed onto it.

The experience

The object

The offered the guests to become “junior naturalists”, an imaginary title that came with a real, physical insignia carved out of local wood. Since the title corresponds to no real-life rank, we tried to give it emotional importance.

Sharing

The children going through our experience were supposed to share the tablet. But “hand the tablet to player one” was not an acceptable way of formulating that instruction. We decided that each participant was going to touch a “spirit stone”, an actual local Native American traditional artifact, to discover their spirit animal. Then, each participant’s spirit animal would, in turn, be called to take part in the experience.

When we tested the experience and I asked the guests how the spirit stone knew which spirit animals was theirs, they said they didn’t know. When I asked them if they would believe me if I said it was random, they were very surprised. For that, I actually credit my script, on which I put a lot of effort.

The script

I wrote the whole script for the experience. While it’s not long, it was very precise. For example, the first version of the spirit animal selection was made while I was away for a couple of days. It was, “You are the hawk”, “You are the stag”, and so forth. That did not seem compelling enough to me. People are not hawks or stags. And why would they be one more than the other? I thought about it and came to the conclusion that the stone should be able to feel a connection between the guests and their spirit animal. That connection should come from a guest’s innate quality. That quality should be general and positive enough that anyone could identify with it. They ended up being (as per the script linked above):

Your inner strength channels the spirit of the stag.
The wisdom in your heart channels the spirit of the owl.
Your clever cunning channels the spirit of the fox.
Your persistent nature channels the spirit of the turtle.
The glee in your heart channels the spirit of the frog.
Your sense of adventure channels the spirit of the hawk.

Everybody hated the frog, but no one could think of another animal that was native to the area.

Similarly, passing the tablet had to be formulated in a different way each time, in order for the action to feel natural and not to feel repetitive:

For this activity, pass the tablet to the stag.
The owl should hold the tablet for this activity.
It’s now the fox’s turn to hold the tablet.
The tablet should be held by turtle for this activity.
Pass the tablet to the frog for this activity.
The tablet should be passed to the hawk for this activity.

Yes, the stag and the frog carry the same instruction in a different order. I hoped that phrasing would still carry enough variety. This replaced a placeholder “It’s the [animal name]’s turn.” That kind of flavor text cannot just inform, it must keep the guest engaged. Ideally, we should have had a different way of phrasing a call to each spirit animal. That being said, the experience as a whole was short enough that one was actually enough. This level of variety gives the impression that the narrator is this mysterious person with an unknown agenda.

We were lucky enough to find a fellow student with an amazing crystalline voice for the narrator. Everything the narrator said sounded ethereal and otherworldly. It added a lot to the abstract quality of that non-diegetic character.

The activities

To launch an activity, the tablet would scan one of five podiums in the park and our avatar, AB Brooks, based on a real-life naturalist, would come up and suggest the activities to the guests. The first ones required active use of the tablet.

The second podium offered guests to measure the angle of the branches of a tree in relation to its trunk. From this, one can deduce how much exposure to the sun the tree has had overall. It was probably the podium that had the most changes in its design during the project’s lifetime.

The three versions, first with small text, then with big text, then with colored arrows.

Unfortunately, I no longer have access to the accompanying interactive interface overlay. It allowed users to interactively position lines on the trunk and the branch and get a reading of the angle.

This activity did not required the use of the tablet at all. It was less high-tech. Still, it is important to remember that enjoying nature without technology was the main goal of this whole experience.

Our trailer

A lot more went on that semester, but those are the highlights of SEECQUEL, along with our video trailer.

This Game is a Business Card, looking back

We’re done with This Game is a Business Card. Looking back, it was even more of a reward journey than I would ever have hoped for.

This Game is a Business Card is made, like most games, of many parts. The user-facing code is written in node, compiled for web-browser JavaScript, but the story is written in Ink. I was very keen on the story being written in Ink, and I was not sure it would happen.

The Tech

Ink

When I arrived, the team really wanted to use Twine but I tried to make the best case I could for Ink. I also explained the basics of the three-act and the five-act structure of storytelling. The team saw that organizing the story around a five-act structure would in fact be easier than not knowing where the story was going to go and they ran with that. None of my individual contributions made it, but the fact that they took my biggest one made me feel hugely included. Then, I really pushed for Ink. I expected them to say know. It is still less known than Twine, but I was absolutely sure it was better. I sent links to videos of its creator, Jon Ingold, and Ink got chosen. Shortly after, the main writer got convinced that was the right decision.

To start with, it’s interactive fiction, my all-time favorite genre. And, not only that, we used Ink, which I still think is the best tool to make IF nowadays. Ink proved itself many times to be invaluable. The two main key ways were:

Arun who’d never coded before found it easy enough to write a story that was interactive. First, it was only somewhat interactive, but he then managed to use Ink’s more subtle abilities with more and more ease.
It was blissfully easy to Integrate Ink into JavaScript. At first, there were a few hurdles getting Ink to work with Node, but once that was done, everything was incredible smooth.
Once we needed some serious strategy, Martin just went into the Ink script and optimized the strategy aspect until it was just right. I never had to adapt the code at all. This is a good thing.
When the story got long, it was trivial to separate the acts into separate files that easily compiled into a single structured story file. This also came from the fact that we had organized the story properly.

Vue.js

I really wanted to use Vue.js as the framework. It is not a game framework, but I didn’t want this to feel like a classic game. It had to feel like something that was more akin to a web app in the sense that it was more immediate. Games have a media layer in a way. They are more removed. Web apps seem more present. You can tell that the text is ready to be selected. It feels just on the surface of the page. I wanted it to be like that. That the buttons were real web buttons. There was also an advantage with the way Vue allows for data binding. Just changing the value of the data would be enough to update the onscreen display. This is much simpler than most game code, and efficiency is nothing to be laughed at.

With this, we got all of the effects we needed. The game was simple enough that CSS transitions provided us with the fades and pans that fit our requirements and, even better, Vue let us access them easily.

The Game

The Objectives

In the end, the game ended up being about running a game company and having to launch a new title under some pressure. It was, after all a promotional game for Tiny Hydra that showed what our services were about. There were four objectives, some more explicit than other:

Make a profitable game
Keep your finances afloat
Keep the team’s morale high
Release on time

I always focus on making a great game and keeping everybody happy. That’s not good enough. “Overly dependent” means I called Tiny Hydra too often. We want to show that we’re useful, but not that we’re the only solution to every problem.

As it turns out, any one of these objectives is trivial to accomplish. Anyone who plays the game once can easily reach all of them. Reaching all of them, however, is not as easy. And making a profitable game at the cost of ruining people’s lives is not an ending that we would qualify as successful. The opposite, people who had fun making a game that didn’t sell is not a great ending either. Can you hit all of those targets? Which platforms should you launch on? PC or console? How about the brand new console that’s not even out yet?

We had a lot of talks about what the next-gen console should be. We decided on the Syga Exodus, which naturally comes after the Sega Genesis.

The Characters

I gave some guidelines to the text and the characters: no text should be only informative, it must always carry the identity of the game with it. And the characters must all have a strong identity. It took time, but I really think we managed to make characters that had identities of their own. I’m particularly fond of Lele, on the right in the illustrations below. “Lele”, or 樂樂 (traditional), 乐乐 (simplified), is based on a Chinese girl I met when I was a high school and took a trip to China. (This is the best way I could get Google Translate to offer a pronunciation for her name.) The name means “giggle.” The way it turned out, our Lele ended up always being grumpy, sort of like Rosa in Brooklyn Nine-Nine. You did not want to mess with her. I really wanted her to say with her deadpan face when she met a new character later in the game, “Hi. I’m Lele. It means giggle.” But everyone else thought it was too on the nose.

A gallery of fun and expressive character really made the game come to life.

The interface

I did everything I could to make the interface follow the instructions in the Ink story. I think I was pretty successful. Often, when something was needed that was simply not there, it could be implemented very quickly, sometimes in minutes, although not always.

There were two statuses that we decided to show onscreen somewhat directly. One was the budget and one was the morate. The budget would be shown in the corner after the concept was brought up in conversation. It would change an icon at the bottom right of the screen. The morale would change the overall illustration of the office. I must credit our artist Yavanna for what I think are amazing illustrations. For me, I did what I always advise people to do: better than clever code, choose the right framework.

Reflecting status changes in game

I have very little to do to reflect status changes in game. All the statuses were kept somewhere in the ink story. Ink allows variables to be watched. When a variable is watched in Ink, I updated it in JavaScript. Then, when a variable is updated in JavaScript, its bound value is reflected in the HTML. That was it. I never needed to do anything more.

For the budget, for example, there were six levels. I had a map that linked each level to a file name for each of the six illustrations. Each time the budget level changed, the JavaScript value of the budget changed automatically. That meant that the file name for the illustrations was automatically updated based on its corresponding value in the map. The main thing is: none of this required any assignment on my part. It keeps the code short and readable. (But still, as always, fully commented and documented.)

The credit and my huge admiration go to the artist, Yavanna Puts.

Geographical difficulties

One aspect that got in the way of the making of this web game was that, while most (but not all) of the team was in the Netherlands, I was in Paris. Still, the team was definitely a mix of cultures. In the Netherlands, there were some Dutch people, some Brits and the man who was running the company was an American. We also had some contributions from someone who is currently living in Germany and who I think is British as well. Creating a multicultural world was one of the goals of the game and while our cast of characters was definitely more diverse than we were, the fact that many of us approached many things differently was absolutely an asset. That said, it would have been nice to be geographically located closer together.

Final thoughts

This Game is a Business Card is by no means the most high-tech game ever made, but it’s still something I am truly proud of. I find it ingenious, lighthearted, and colorful. It carries strong meaning through gameplay, which many games struggle to do. It has both good content, good mechanics, and a good interface. I feel like I took part in all of that, but that I am not fully responsible for any of it. That’s a good thing. Good games come from collaboration. If one feels that even one aspect of the game comes from one person alone in their tower making it up on their own, something is probably wrong.

The fact that the tech was lightweight allowed us to keep the development process nimble and to make changes quickly all the way to the end. The founder of the game studio Inkle, Jon Ingold, praises text for that reason. His current games are not presented as graphical, but their contents are still based in text. They can be modified quickly to keep the design flowing. It does not mean that games that require complex and heavy production are the wrong way of doing it. Just that there is true value, I think, in also, sometimes, keeping the methods of production fluid and flexible.

Beyond the Stage, in retrospect

Beyond the Stage was my project at the Entertainment Technology Center at Carnegie Mellon University for the Spring 2012 semester.

Beyond the Stage was one marathon of a project. Most team members remember that it didn’t go ideally, but I still remember it very fondly. I loved what our goal was supposed to be. Entertainment technology and the stage working together. In the end, it turned out that even without the problems that were out of our control, the project was completely overscoped from the start. Still, many great lessons were learned on the way.

From Powerbomb the game based on the Pulizer-nominated play by one of our clients, Kris Diaz, we had a crash experience-based course on tough game design and design documents. We learned a lot about how to make an interface clear and how to manage player expectations and allow them to master the gameplay.

At my suggestion, we got to use the Impact game engine, which was for all of us our first time working with HTML5 and the Canvas element. Most of the programmers on this project really did love Impact and are still using it for some of their personal projects.

For the Vera Stark part of the project, for our client Pulitzer prize winner Lynn Nottage, I got to dive deep in Scala and the Lift Web Framework. It was hard, but I now understand and love them.

I also wrote and edited (but did not film) our team promo video:

Special thanks to Evan Brown, Rayya Brown-Wright, Brad Buchanan, Josephine Tsay and the oh-so-unique-and-irreplaceable Dana Shaw for being the best team members I could ever have wished for.

Heidegger, in retrospect

My last project at the Entertainment Technology Center at Carnegie Mellon was Heidegger, at our Silicon Valley campus. The client was Electronic Arts and we were very lucky to be able to work with several people from the Dead Space team at Visceral. Our project was about finding a way to deduce player types though gameplay analytics and from there predict how likely people will be to like another game.

I worked on the interface with the very talented Anabelle Lee. She took care of aesthetics and I was in charge of functionality. I took that opportunity to improve my skills with the HTML 5 Canvas and jQuery. But, most of all, I got to experiment with Websockets and the fascinating Tornado web server, written in Python.

I also wrote and edited our promo video. I did not film it, though. That part went to other members of our team.

I would like to take the opportunity to thank Ben Medler, our fantastic client contact for just being so awesome.

A webserver optimized for websockets

As part of my OCCO project, I made a web server that makes the use of websockets easy. I din’t make it from scratch, of course, I made it with Tornado. Tornado is a very lightweight, flexible and extensible web server written in Python. It makes the use of websockets trivial.

My whole code is on GitHub:

https://github.com/eje211/heidegger

This site is a mockup of a modification of the interface of Electronic Arts’ Origin game distribution system. My team added a panel that informs the player of their Bartle player type. My websocket feature allows me to change the player type in real time during presentations. This was really made to have a good reason to use websockets more than anything else. The Javascript part of websockets never changes. It’s the part that’s on the server that’s often problematic. In this case, I have a page with the player type chart and one or more pages with a remote controller to change it. The code to handle this is absolutely trivial:

CHARTS        = []
CONTROLERS    = []

class ChartHandler(tornado.websocket.WebSocketHandler):
    '''
    Set up a connection to the chart. We won't be receiving any
    information from it, just sending some.
    '''
    def open(self):
        CHARTS.append(self)
    def on_message(self, message):
        pass
    def on_close(self):
        try: CHARTS.remove(self)
        except ValueError as e: print('Could not remove chart handler:', e)

class ControllerHandler(tornado.websocket.WebSocketHandler):
    '''
    Get every change from each controller and send it back to all the other
    controllers, if there are any. That way, if other controllers make a
    change, it will include the ones made by other people.
    '''
    def open(self):
        CONTROLERS.append(self)
    def on_message(self, message):
        for chart in CHARTS: chart.write_message(message)
        for cont in filter(lambda x: x != self, CONTROLERS):
            cont.write_message(message)
    def on_close(self):
        try: CONTROLERS.remove(self)
        except ValueError as e: print('Could not remove controller handler:', e)

I keep a list for each of the two types of websocket handlers and keep them updated as the sockets are connected and disconnect. And, as message come, I just pass them along to the relevant pages. That’s really all there is to it.

Tornado also requires me to deal with the delivery of files manually. That was new to me, coming from Apache, Lift and Zope. It was interesting and made me aware of many details.