Today, we spend as much time immersed in the digital world as we do the real world. But our computer interfaces are flawed. They force us to work in 2D rectangles instead of our native 3D spaces. Our brains have evolved powerful ways of processing information that depend on dimensionality — and flat screens and operating systems short circuit those tricks.
The result is a bit like reading War and Peace one letter at a time, or the classic example of a group of blind folks trying to define an elephant from the one piece they are touching. We’re missing the big picture, missing all the spatial systems that help us understand — the space is just far too limited.
All this, however, is about to change in a big way.
With augmented and virtual reality, we’ll be able to design user interfaces as revolutionary as those first developed for the Xerox Alto, Apple Macintosh, and Microsoft Windows. These new 3D computer interfaces and experiences will help us leverage dimensionality, or space, just as we do in the real world — and they’ll do it with the dynamic power of digital technology and design.
How We Naturally Use Space to Think
Ultimately, humans are visual creatures.
We gather a large amount of information about our world by sight. We can tell which way the wind is blowing, for example, just by looking at a tree. A very large portion of our brains, therefore, is dedicated to processing visual compared to linguistics.
If you take your hand and put it to the back of your head, that’s about the size of your visual cortex. Driven by this large part of our brain, we learned to communicate via visual methods much earlier than language. And even after we formulated words, our language remained very visual. In the representation of letters on a spatial grid to create meaning, for example, it’s the underlying spatial structure that ultimately allows us to read.
Space is therefore very important to humans. Our brains evolved to be visual in 3D spaces. We are constantly creating a spatial map of our surroundings. This innate process is called spatial cognition; the acquisition of which helps us to recall memories, reveal relationships, and to think. It is key to sensemaking.
Spatial memory is, in effect, “free.” It allows us to offload a number of cognitively heavy tasks from our working memory in the following ways:
Spatial Semantics: We spatially arrange objects to make sense of information and its meaning. The spatial arrangement reveals relationships and connections between ideas.
External Memory: The note on the refrigerator, the photo of a loved one, or the place you always put your keys are all examples of how space compensates for our limited working memory.
Dimension: Dimension helps us to immediately understand information about an object. For example, you can easily tell the difference between War and Peace and a blank piece of paper just by looking at the two objects.
Embodied Cognition: Physically interacting with space through proprioception — this is the human sense that locates our body in space — is essential to understanding its volume, but it also helps us process thought.
How We Interact With Computers Today: The Graphical User Interface (GUI)
In 1973, Xerox PARC was a hotbed of technological innovation, and Xerox researchers were determined to figure out how to make interacting with computers more intuitive. Of course, they were fully aware of the way humans use visual and spatial tools.
People had a hard time remembering all the specialized linguistic commands necessary to operate a computer in command-line interface (think MS-DOS). So, researchers developed an early graphical user interface (GUI) on the Xerox Alto as a way to reduce the cognitive load by providing visual/spatial metaphors to accomplish basic tasks.
The Alto used a mouse and cursor, a bitmapped display, and windows for multitasking. No one thought the job was complete, but it was a great first step on the road to a simplified user interface. And most operating systems today testify to the power of GUI.
The Problem With Modern Operating Systems
The problem is 2D computing is flat. GUI was invented at a time when most data was linguistic. The majority of information was text, not photos or videos — especially not virtual worlds. So, GUI organization is based on filenames, not spatial semantics.
The resulting “magic piece of paper” metaphor creates a very limited sense of space and prevents development of spatial cognition. The existing metaphor:
- Limits our ability to visually sort, remember and access
- Provides a very narrow field of view on content and data relationships
- Does not allow for data dimensionality
This means the user has to carry a lot of information in her working memory. We can see this clearly in the example of the piece of paper and War and Peace.
In modern operating systems, these objects look dimensionally to be exactly the same because they are represented by uniformly similar icons. The user has to know what makes each object different — even has to remember their names and where they are stored. Because of this, modern operating systems interrupt flow.
Multiple studies have been focused on interruption cost for software engineers. It turns out that any interruption can cause significant distraction. Once distracted, it takes nearly half an hour to resume the original task. In our operating systems, every task switch interrupts flow. The average user switches tasks three times a minute. And the more cognitively heavy the task switch, the more potent the interruption.
The Solution? The Infinite Space of Augmented and Virtual Reality
But now augmented and virtual reality are emerging. And with them, infinite space.
The good news is that spatial memory is free, even in virtual spaces. And allowing users to create spatial systems, such as setting up tasks across multiple monitors, have been proven to dramatically improve productivity by up to 40%.
So, how do we create a system that capitalizes on the opportunity?
The most obvious place to start is to enable development of virtual spatial semantics. We should build 3D operating systems that allow users to create spatial buckets to organize their digital belongings — or even better, allow users to overlay the virtual on their pre-existing real spatial semantics. People already have well established real world spatial memory, so combining the two will lead to even better multi-tasking.
For example, I have a place where I always put my keys (external memory). If I need to remember something the next time I leave the house, I leave it in that spot next to my keys. If I could also put digital objects there, I could become immensely more productive.
Further, adding digital smarts to spatial semantics, users can change the structure dynamically. I can arrange objects in a certain way to find specific meaning, and with the touch of a button, instantly rearrange the objects into a timeline, an alphabetical list, or any other spatial structure that would help me derive meaning — without losing my original, valuable arrangement. Sanddance at Microsoft Research (Steven Drucker, et al) and Pivot by Live Labs are excellent examples of this type of solution.
And finally, the introduction of the z-plane enables digital object dimensionality.
By applying sort parameters to the x- and y-axis, virtual objects take on meaningful dimension. But unlike the real world, where objects follow the rules of Euclidean geometry, in the virtual world dimension can be dynamic. The sort methods applied can quickly be easily swapped out, depending on the user need, to quickly and effectively change the dimension of the objects — allowing users to tell at a glance what is more or less pertinent to a query.
Dimension also creates opportunities for virtual memory palaces.
Memory palaces are a mnemonic device from ancient times that enhance memory by using spatial visualization to organize and recall information. The subject starts with a memorized spatial location, then “walks” through the space in their mind, assigning a different item to specific features of the space. Later, to remember the items, the subject again “walks” through the space and is reminded of the items along the way.
With the advent of virtual 3D spaces, the same type of memory device can be created to allow users to organize, remember, and delineate large amounts of information with the added benefit of a digital “map” of the space; a map that can be dynamically rearranged and searched depending on the user needs in any given moment.
We Are On the Cusp of Another Technological Revolution
Humans are indeed visual creatures, and augmented and virtual reality is geared to help us use those abilities in the digital world. Through these technologies we can alleviate the heavy reliance on working memory needed to operate our tools, and enable instead our natural spatial cognitive abilities. We are on the cusp of another technological revolution — one in which we create superhumans, not supercomputers.
In her 20-year design career, Jody has created just about everything from holograms to physical products and R&D for over 300 companies. She’s spent the last three years working on AR/VR, most notably as a principal experience designer on the HoloLens Project at Microsoft and principal UX at LEAP Motion. Previously, she co-founded and directed Kicker Studio, a design consultancy specializing in Natural User Interface and R&D for companies including Intel, Samsung, Microsoft, and DARPA. You can learn more about Jody’s work here follow her @nothelga.
To get updates on Future of Virtual Reality posts, sign up here.
