A TUI Blog

Out To Launch is a Tangible User Interface that teaches users some basic things about the inner workings of a computer.  Specifically, after playing around with the interface, users will have a better understanding of the finite nature of a computer’s memory resources. The hope is to get them to understand why running lots of programs at once can degrade machine performance and response. This blog is a record of the creation process of Out To Launch, from conception to implementation.

The Good

• Lab format of the class was great. I liked the fact that we got to work on our projects under the direction of the teachers, instead of putting lots of work in outside of class and then getting feedback. Being gently guided during the design process was very helpful. I felt like we were continually making progress.
• Trips to the Museum of Science were helpful and fun. It was interesting to get a behind-the-scenes look at how exhibits are put together. The amount of work that goes even the smallest exhibits is amazing.
• It was very satisfying to have the project come together at the end and to finish the class having created something tangible, something brand new. The potential for publishing about our project is even more exciting.
• User testing at the end was interesting. We got a lot of stuff right, but we also did some stuff wrong (for example, one of our testers was about 5 feet tall and had some difficulty reaching the “execute” button).
• Going through the entire product development life-cycle, from design to implementation to final prototype, was a unique and gratifying experience.

The Bad

• The pace/workload seemed a bit uneven. For the first 90% of the semester, I felt like we were bit-by-bit moving toward our goal of implementation. We discussed TUIs, we outlined existing TUIs in TUIML, we designed TUIs, we critiqued them, we redesigned, etc. Then the last 10% felt like a mad dash to get everything done on time. Put another way, up until about a week before our first in-class demonstration of functionality, I felt that the workload for this class had been significantly lighter than all my other classes. However, from that point until it was time to turn in our written documentation and our personal reflections, the total amount of time I spent working on stuff for this class easily surpassed the total time spent on work for all my other classes combined. Luckily, my two other classes were ramping down in workload, but I shudder to think what would have happened if I’d had big projects to do in them as well. That is not to say I found the drawn out design process useless–on the contrary, I found it incredibly helpful. It made the implementation of our project much easier and more straightforward. I think what I’m saying is that perhaps starting the implementation stage a little bit sooner in the semester would have made the last weeks a little less frenetic.

Truthfully, that’s it for the bad. I had a lot of fun in this class, so there isn’t/wasn’t much to complain about. My favorite thing about it is that, if someone asks me what I did in TUI this semester, I actually have something to show for it–something tangible.

The Final ProductOur final prototype of the project is called Out to Launch!!, because that’s what a user does–he launches programs. A user looks at the program key at the top of the board, places a code sequence one the board, and hits “execute”. Like magic, that program opens, and an invisible user starts to interact with it. For example, load the code for mspaint on the board, and watch in amazement as a smiley face is drawn. How on earth did we accomplish this? Read on to find out…

Implementation in 3 Acts

Act One: The camera

The project relies on computer vision. The camera (a QuickCam Pro 9000) is placed underneath the board so it can read the code on the bottom of each token (those round things in the picture). It follows, then, that our board must sit on a glass table so the computer vision can do its job. So the first part of the implementation system is a glass table and a webcam sitting the floor looking up at the Out to Launch!! board.

Act Two: The board

The board itself was made from laser-cut acrylic, compliments of the design guy in the group, Nate. The red sheet has the holes cut out that hold the code tokens themselves. The yellow sheet has smaller holes cut in it so that the webcam can read the codes from the bottom, underneath the table. Each token is color-coded and coded numerically, providing maximum flexibility for the user. We used the cut-outs from each piece of acrylic to create the tokens–the pieces from cutting holes the red sheet hold the topcodes on them, and the pieces from the yellow sheet act as handles that allow for easy manipulation of the code tokens. Fabricating the board itself was tricky, because the dimensions had to be absolutely perfect. Computer vision can be finicky, so if the holes cut in the yellow piece weren’t big enough, or if they were off center, the entire system wouldn’t work. So, hat’s off to Nate for his precision laser-cutting. Additionally, there is a convenient bin at the bottom of the board that holds all the tokens that aren’t in use.

Act Three: The software

The software component of this project has two parts: Java and AutoIt Script (AIS). The Java component consists of Mike Horn’s topcodes webcam library combined with a few custom classes that use computer vision to read topcodes on the bottom of each code token (found here). The group of topcodes returned after the computer vision is invoked is then parsed into program codes and checked against the valid program codes loaded during start up. If the codes are valid, the Java code makes a call to exec() and the appropriate program runs. Just opening a particular program is boring. We wanted to simulate actual user interaction, so we used AIS, a Windows GUI scripting language–this is what simulates an invisible user that interacts with each program after its open. When Java executes each program, it’s actually running a compiled AIS file. Each program uses its own AIS file to open. For example, the AIS that is executed when a user loads the “notepad” code sequence first runs “notepad.exe”. It waits until the notepad window has focus, then sends a sequence of keystrokes to the new document, simulating a user opening notepad and beginning to type. To get the AIS scripts to execute from the Java code, I added the path where the executables live to the system’s PATH variable.

Introduction to Tony

Our project is unofficially called Tony. No clever acronym–just a name (we’ll pick something more relevant later down the road). It’s basically an interactive board that allows a user to run programs in a construct that simulates the limited resources of a computer’s memory. The purpose is to demonstrate to the user how the amount of RAM in a computer can limit the number of tasks it can perform in a tangible way. In other words, without enough RAM, Tony is more cumbersome to use. Add some RAM (within Tony’s construct), and he becomes easier to use–much like that old Pentium 4 you have chugging along with 512mb of RAM, but which would benefit enormously from a 2gb RAM upgrade). Today, computers are ubiquitous. You’d be hard-pressed to find a person who doesn’t come into contact with computers on a daily basis. What’s surprising, however, is how little most people know about the fundamental concepts behind a computer’s operation. This information, though interesting for its own sake (at least, this author thinks so), is very important when someone is in the market for a new machine. Specifically, this project attempts to instill in the user an understanding of how RAM works and why more is usually better–but not always. After playing around with Tony, it’s our hope that a user will be able to make a more informed buying decision when next in the market for a new computer (and it doesn’t hurt that the user learned something in the process!).

Design Process

Our design process was very much an iterative one. We took a core concept–the idea that with limited memory, a computer must work harder to perform the actions expected by a user–and started with the most basic design possible. Initially, the vision was a flat board with a designated square on it that represented the program “in focus” (that is, the program the user is currently interacting with). Each program is represented by a tile, the size of which depends on the program’s RAM footprint. Photoshop, for example, would have a much larger tile than would a program like Notepad. To interact with our computer, the user would have to place a tile on the RAM board, where it can be moved only by sliding. As more programs are opened, the space on the RAM board starts to become cluttered. Consider the following scenario:• A user is interacting with Notepad (that is, the Notepad tile is in the “focus” square).• There are several other programs “open” (that is, other program tiles are on the RAM board’s surface)–iTunes, Photoshop, Firefox, Norton Anti-Virus, and Outlook.• The user wants to change the song playing in iTunes. In order to do so, he must move the Notepad tile out of the focus square and move the iTunes file into the focus square.• Depending on the position of the tiles, this seemingly simple action could take a lot of “work”, as the user may be forced to rearrange the other tiles on the board in order to make room for the Notepad tile to leave the focus square and for the iTunes tile to enter it.In this way, the user sees that the less RAM (ie surface area on the RAM board), the more work he must do in moving programs in and out of the focus square.

Design decisions

In conceptualizing this project, we were faced with several issues:• Is there a task for the user? Or, does the user do his own thing?• How do we prevent a user from picking up tiles off the board (and thus defeating the purpose of the project)?• How do we force the user to actually do the work of the processor? This involves decisions about the shape of the RAM board that would prevent the user from simply lining all the tiles up against one side (which would represent very efficient memory allocation, but would, again, also prevent the user from having to do the work of the processor. Currently, we’ve decided that providing a task for the user would demonstrate the concept more effectively than making it a free-form interface and hoping he stumbles upon the concept. Ideally, we’d have a computer connected to the board that runs programs according to what’s on the board. So, if a user wanted to open iTunes, Microsoft Word, and Photoshop, he’d have to place the iTunes token on the board. To bring a specific program into focus, he’d have to move that program’s token into the focus square. In this way, a user would utilize the RAM board to interact with the computer and complete an assigned task.To prevent a user from simply picking up a token and moving it, we’ve designed the board with a clear plastic covering, so that a program token may only be moved with a “processor token” (which, at this point, we envision to be a magnet, thus making the program tokens themselves magnetic as well).The third issue was solved (at least for the time being) during a discussion in class about the pyramid-shaped memory hierarchy. We figured that if the board were triangle-shaped, with the focus square (or “cell”, as it will be referred to from here on out) at the top, then a user would have to play a game of token shuffling before he would be able to put the appropriate program token in its place at the top and use the program.

Where do we go from here?

From here, we must continue to refine our ideas about how a user will interact with the interface. What, specifically, will we ask a user to do? Play a song, open a .doc file, and crop an image? What attribute of our program tokens will we use to impress upon a user the difference in RAM footprint of various programs? How will we make it more difficult to move a RAM-hungry program (ie Photoshop) as opposed to its opposed (Notepad, perhaps)? Do we even want to differentiate between RAM footprints of programs? Is it enough that the user simply understand that RAM is needed to run programs, and that it is a limited resource?Following these questions will be questions of implementation. How will we differentiate between different program tokens?• Weight?• RFID?• Computer Vision?Further, how will we arrange the program tokens on the RAM board? Will they be in a “snap-to-grid” fashion? This issue also effects how we design the tokens–if we decide to display RAM footprint by making the program tokens different sizes, will a larger program simply take up more space, or will it occupy more “cells”?

Looking Forward

Although it’s obvious there are still many decisions to be made, I feel this project has a lot of potential. The design issues are by no means insurmountable, and I think given lots of discussion and feedback, we can turn Tony into a very useful museum exhibit (or Apple Store exhibit!).

When tasked with designing a tangible user interface, the first question I asked myself was, “To do what?” It was difficult for me to come up with a interface without first knowing what the purpose of it would be. A drawing program? A film-editing program? An urban-planning program? Designing an interface without knowing how the it will be used is like making clothing without knowing who will wear it, or like building a boat without knowing who will be captaining it. You could do it, but your final product would probably suck equally for all tasks. With that in mind, I tried to change my thinking and first thought about what program would be interesting to implement. After our museum trip, it was made apparent that our projects should be educational–they should teach the user something as he’s using it. So at this point I have two points to consider–what would be fun to build and educational at the same time? I came up with three ideas. The first one is an exhibit that helps users understand, in a very general way, what an operating system is. I picture an interaction surface with shapes cut into it–one for each operating system we’re learning about. Let’s say linux (representative of all distros), windows (again, representative of all windows versions), and OS X (call it 10.4 and 10.5). Next to this surface are various blocks of varying shapes, representing programs. Each block can only fit into one shape, the same way many programs are compatible with only one operating system. Then we have a squishy block that can fit into each operating system–Java. In this way, a user understands that certain programs are built to run specifically inside certain operating systems, whereas others are created with more flexibility to run in multiple OSes. So a user learns that trying to make a program run in an operating system other than the one for which it was written makes just about as much sense as trying to put a square block in a round hole.

We went to the Museum of Science on Saturday. We met with several employees at the museum and learned what goes into making a successful exhibit. What struck me most was the differing views of the staff on what makes an exhibit successful. The first woman we met felt that the entire point of a museum was “informal learning”, where a patron learns something without even realizing it. From her perspective, regardless of how popular an exhibit is, if it’s difficult to say what a person is learning from it, it’s not a good exhibit. This directly contrasts with what a guy we met later in the day said. His point of view was that a successful exhibit is something that provides an experience the user can’t get at home. He felt that an exhibit that attracted lots of eyeballs was a successful exhibit. It seems to me that the correct answer lay somewhere in between. An exhibit can only impart informal learning if it draws people to it. So a certain measure of wow-factor is necessary to bring people in. Once they are engaged, informal learning can take place. So, given the two extremes–the informal learning, where a museum’s sole purpose is to educate its patrons, and entertainment, where a museum’s purpose is to draw a crowd, I think a truly successful exhibit entertains just enough to draw a person in, but not so much that they’d sit there for hours in one exhibit. It teaches just enough that a person comes away with more questions about his/her topic of interest, rather than bored and annoyed from too much information being spewed in his direction.

For our latest assignment, we were instructed to model the TUI we had previously written up in TUIML, which is a modeling language designed specifically to help developers wrangle with TUI concepts. My initial impression is a mixed bag. On the one hand, it is definitely convenient to have a way to abstract implementation away and just focus on what the interface will actually do. Doing this allows us to take a “don’t sweat the small stuff” mindset and prevents any premature hang ups concerning implementation. On the other hand, there were several times when I felt that the language may be too broad, or not specific enough. For example, in the interface I was modeling, A Physical Approach to Multimedia Stories, most of the interaction came when a user moved a small selection tool onto various areas of an interaction surface. These parts were designated by projected images and keywords–things that weren’t quite physical, but were critical to the functioning of the system. A token is described as “a graspable object that represents digital information or a computational function.” These projected images weren’t “graspable”, but interaction with them did represent various manipulations to the behind-the-scenes digital information. Likewise, a constraint is a physical object that limits the behavior of a token. The interaction surface is certainly a constraint, but what do i classify these projections as? Part of the surface? That was my sticking point in the usage of TUIML.  I’ll expound on this later, but for now I have to go to class…

Wikipedia defines a TUI as “a user interface in which a person interacts with digital information through the physical environment.” Going by this definition, I’d consider a mouse and a keyboard parts of a TUI–after all, they are part of the physical environment and we use them to interact with the digital information. My idea of TUI goes a step further–it’s a user interface in which a person interacts with physical representations of the digital information. Taking URP as an example, we see that a building is represented by an actual physical structure that resembles a building. Wind also has a physical representation, as does the angle of the sun (it’s a clock!). In the previous post, I mentioned the Tangible Viewpoints Interface. Each character in the story was represented by a little pawn-shaped object, something you could pick up and hold. In this way, tangible user interfaces give the user a more familiar and often more intuitive method of interacting with the information on the screen.Of the TUIs we’ve looked at so far, the one I liked the most was the I/O brush, which allowed a user to swipe the brush across the surface and “borrow” the color and texture from that surface. He could then “paint” on the screen with that information. For example, if you were using the brush to draw a circle on the screen and you wanted to color it green, you’d have to swipe it across something green–maybe your friend’s shirt. I liked this one best because it seemed to add an extra dimension of interaction to the task the user was performing, instead of simply replacing the old mouse and keyboard way of doing things.

Our first assignment was to read a TUI paper and prepare a brief presentation on it. Mine was on the paper located here. It was a “Tangible Viewpoints Interface”, designed to let a user immerse himself in a story told from multiple viewpoints. The author would write a story from multiple viewpoints and then chop those viewpoints up into story segments, each referenced by keywords. The tangible part consists of an interaction surface that looks a lot like a chess board, complete with clear pawn-shaped pieces, each of which represents a specific viewpoint within the story. Keywords are projected on the surface. The closer a keyword is to a specific character (pawn), the more relevant that story segment is to that character. If a particular keyword is equidistant from two (or three or four, etc) characters, then its story segment involves both (or all). A user navigates the story by sliding a small convex “lens” around the playing surface, letting it rest on whatever keywords he feels like. At that point, the appropriate segment of the story is played on a display screen in front of the user. Segments can be sounds, video, images, text, etc. In this way, a user can control the depth of character development in the story by selecting keywords.It’s a pretty interesting idea, but as I read through the paper, I started wondering: Why go to all this trouble? This stuff could just as easily be emulated on a screen–instead of building a physical model, just write a program. Then a user could simply click on the keywords of interest, and the appropriate story segment would play. In class, arguments in favor of this tangible interface were the facts that it facilitates collaboration and it may be more intuitive for a child. However, I’m not convinced. At this point, collaboration can easily be achieved through a networked version of a program, and children are more computer-savvy than ever. So it looks like one of the questions I will be trying to answer during this course is, “Are tangible user interfaces preferable to graphical user interfaces? If so, in when?” Stay tuned and maybe i’ll find an answer.