Interview with Pavel Barter from PC Zone Magazine in UK

[ Pavel ] How would Diamond Touch enhance someone's enjoyment of a game like Warcraft III?

[ Ed ] In traditional console games (e.g., XBox, Playstation) you are forced to look away from other people to face a television set. People's hands are tied to the controller thus producing very little awareness information. Diamond Touch is a table top interactive surface that allows multiple people to play in a face to face setting. Thus people can see the game and the other people around the table. They can observe the rich hand gestures of others over the digital surface and they can hear the verbal utterances made through speech commands. The rich hand gestures provided by the Diamond Touch provide an engaging feeling normally only provided by manipulating physical objects such as a gun in an arcade. In one example, we show two people using whole hands to pick up a digital table in The Sims and in another we show a tree stamping gesture using a fist. These interactions make Diamond Touch a viable technology for repurposable home gaming.

[ Pavel ] Despite gaming's interactive nature, do you think that games could still be far more immersive, and do peripheral devices allow the opportunity for this?

[ Ed ] I don't consider playing a console game over a television with a game controller immersive. Arcades have done a lot to make gaming more imersive but they require expensive specialized hardware that would simply not be practical to have in a home setting. Peripheral devices such as flight joysticks and racing wheels are expensive and only work with a limited number of games. Also, since game designers cannot expect everyone to purchase and use these specialized peripheral devices, game interaction is still predominantly designed for the game pad. Hopefully digital tables will help to bridge the gap between home console and arcade games as they support rich hand gestures in a repurposable way.

RE: Buying a Digital Table for My Home

A question from Rainer Schuller

[ Rainer ] I´m a huge fan of RTS Games and just saw ur W3 Video. First of all: Wonderful Idea! Really love it! I would be very interested what your opinion is about the release date and the approximate cost of such a Command Panel?

[ Ed ] Large display touch technologies are already available from commercial companies such as Mitsubishi and SMART. Currently, these input technologies are expensive ($9 - $20K USD) but their price will no doubt drop with time, remember that you can get portable computers today as powerful as computers 30 years ago that were the size of large rooms and would easily cost millions of dollars. Realize that these prices have already dropped about $10'000 USD over the past 5 years and are expected to continue to drop for many years to come.

Bill Buxton has stated in an interview that "What is going to happen In the next five to seven years is very high resolution displays, that is as many pixels per inch on the display as you have on your laptop screen. They are going to be cheaper and thinner, but cheaper per square foot than the whiteboards that are on your walls today. I repeat, to get a really, really, really high resolution large display the size of the whiteboard in your classroom or your office is going to be cheaper than the whiteboard that you have there currently. And it’s going to be 6 mm thick. That’s a little bit thicker than a piece of paper and it’s going to cost around $10 a square foot. And it’s going to transform completely how we think about information and information displays and how we interact with them."

When asked who is doing that now Bill responds: "There are a number of companies, none of them Canadian, doing the basic technology. Phillips is doing a lot. Some of the core technology is being done in Cambridge England by company called Cambridge Display Technologies. So there’s a lot of materials, it’s very expensive stuff. There’s a lot of cooperation with companies like Sharp and Mitsubishi and other companies in Japan. But I don’t care about the technology, I just know it’s coming. The question is, who is going to be smart enough to know what to do with it once it arrives? And that’s where there is an Alberta company called Smart Technologies in Calgary which is the world leader in electronic whiteboards."

Bill is probably right, we should expect to see stuff like this in homes within the 5-7 year mark at incredibly affordable prices. However, in order for such technology to succeed, there needs to be advances in display hardware, input hardware and large display interaction software. My research is specifically targeted at the software domain.

Hopefully all of these things will come together and you'll be able to enjoy tabletop games and applications in your home in the near future. From the science fiction author William Gibson "The future is already here - it's just not uniformly distributed."

Two new videos added

I've recently added videos of some previous work on table tops. In particular, the Shape of Conversation video shows multiple people doing crazy things with speech on a table top. Also, the GroupLab Diamond Touch Video shows work by Rob Diaz-Marino using the MERL Diamond Touch Hardware.

These videos should give you a bit more background of how we got to using Speech and Gesture interaction over a table top.

Please feel free to post comments and suggestions! Thanks!

RE: Lag in the Scrolling Gestures

A question from Rob "Xemu" Fermier regarding the lag in scrolling gestures.

[Rob] Very cool technology and a nice approach for demonstrating it. Using contemporary examples like Google Earth and Warcraft 3 is an excellent way of taking relatively abstract concepts and making them real for people. The gap between academia and "real world" software development is often pretty huge and it's great to see more approaches like this that can bridge that gap.

[Ed] Thank you. One of the things that I like most about my research is that computer games such as Warcraft III allow it to appeal to a larger audience. That is, my research is not only of interest to academics, but to the general public.

[Rob] I noticed the frame rate for scrolling, etc hitching a bit ... I'm curious if that was an artifact of the technology used to do gesture detection, or just the machine playing the game?

[Ed] Diamond Touch is a special type of input device for tables that can detect the gestures and movements of up to four people simultaneously. This input currently runs at a frame rate of 30 frames per second which does not seem like a lot but they are more than sufficient for gross gestures such as using a whole hand to pan a map. Also, since four people can interact simultaneously the effective frame rate is really 120 frames per second. Modern windows applications will often respond to the mouse at a rate of around 120 per second. This means that there is a bit of jerkiness in the Warcraft III panning gesture. This could be resolved by using interpolation of mouse events between frames. This is done in the Google Earth demo, thus the jerkiness is almost non visible.

Certainly the limitations of today's tabletop technologies would make it difficult to play Warcraft III as well as you can with a mouse and keyboard. But eventually, these limitations will be overcome and we will be able to interact with computer games in ways that were previously not possible. I detail some of the possibilities in a recent paper published at Pervasive Games 2006. It's important to realize that tabletop games are not replacements for mouse and keyboards over Warcraft III. Rather, tabletop games represent a new genre of gaming where people can interact face to face rather than having to look away from each other as we do with current console games. Being able to interact with rich hand gestures and speech provides an engaging experience that normally can only be found when manipulating physical objects such as a gun in an arcade.

The goal of this research is to understand the capabilities and limitations of speech and gesture tabletop interaction. This will hopefully inform the design of future multimodal tabletop games.

RE: It doesn't game as well as a keyboard and mouse

I've been recently reading some forums commenting on work that I did at Mitsubishi Electric Research Laboratories. Most of them are postive (there's usually a couple 'If your system could play some game I'd wet my pants' type comments) . A few conspiracy theorists seem to think that my demo is too good to be true - and someone calls me to task on my Warcraft skills.

First, I'd like to dispel the myth: this work is NOT fake, if you'd like to see exactly how we built this system please read our research paper. When you look under the hood you'll see that this system is really just a simple conversion of gestures and speech to standard keyboard and mouse commands.

Second, people commenting about how this system is not as good as a keyboard and a mouse are totally missing the point of this research. It's not about being more efficient than a keyboard and mouse but rather this work is about making actions public so that others can
double check to ensure the best outcomes.

Many things in life are not like WarCraft III where you can die and play again. Think about safety critical applications such as real life military command and control or air traffic control. Here the collaborative decisions have a direct impact on people's lives. By making actions public on the tabletop others can monitor your activity and ensure that you are doing the right thing.

Warcraft III is really designed as an example of a military command and control situation rather than a replacement for the keyboard and the mouse in the game.