Reality Based Interaction

A New Style of Interaction

At the moment we are seeing emerging a new style of human computer interaction.
Traditional mouse and keyboard interfaces have been replaced by new methods of interacting with a computer that are less bound to a desk.


These new interfaces have been defined as Post-WIMP. Replacing the Windows, icons, mouse and pointer paradigm. They include a d
dizzy number of interface styles
  • virtual reality
  • augmented reality
  • tangible interaction
  • pervasive computing
  • context-aware computing
  • affective comp

Reality Based Interaction

Robert Jacob has unified all these diverse styles of interaction under the title Reality-Based Interaction and he proposes that the unifying features is:

We believe that all of these new interaction styles draw strength by building on users’ pre-existing knowledge of the everyday, non-digital world to a much greater extent than before.

The idea of Reality-Based Interaction is that the new styles of interaction allow us to use skills that we already have from interacting with the real world. Jacob proposes that there are 4 classes of skills that Reality-Based interfaces can use:

Reality based interaction can use our knowledge of how the physical world works. This is called Naive Physics, because it doesn’t rely on us knowing the mathematical laws of Physics, just our normal understanding of the physical world

Tangible interfaces make it possible to interact with computers using physical objects, making use of our knowledge of their physical properties. The urban planning demo at the end of this video is a good example.

The ReacTable is an other example that uses more abstract interaction via physical objects.

Interfaces can also use our natural understanding of our own bodies and our ability to move our bodies.

New gaming interfaces such as the Wii, or the not yet released Natal system make use of bodily interaction to enhance the immersiveness of their gaming experience/

The film Minority Report made popular the idea we could interact with computers using natural physical gestures, an concept that has been explored in research for a while, e.g. the system below

Gesture interface
Body Navigation (Full Documentation) from ole kristensen on Vimeo.

Bodily interaction has also been used as a way of creating virtual performances in which performers intact with digital media.

Environment awareness is about creative interfaces that use our normal awareness of a physical environment and our ability to move around it and interact with it.

Pervasive and location based gaming embeds digital content and interaction at particular real world locations spread across cities, making use of our prior knowledge of our urban environment and how to navigate as a means of interacting with a game. The work of the group Blast Theory is a good example of this.

Virtual and augmented reality systems create digital environments that we are able to interact with as if they were real.

Finally, there are also interfaces that use our social skills that we use to understand and interact with other people.

Tangible interaction systems like the ReacTable are easy to use collaboratively and socially, as this collaborative performance shows. They leverage our ability to cooperate with physical props.

Boundary functions is an installation that makes explicit an aspect of our social interaction: social space. It encourages people to play with and think about their spatial interactions with others.

Finally there are many researchers, including myself, have been working on getting computer systems and virtual characters that are capable of social interaction with humans. The Natal Milo demo is a (largely smoke and mirrors based) example of this. For more information on my work in this area see my research pages.
PDF Download    Send article as PDF to