In the recent years, we have been witnessing a staggering progress in smartphone technology. In addition to net surfing, listening to music, watching movies, reading books, and other old-school activities, smartphones now allow us to use augmented reality apps. They enhance the image of our surroundings as captured by the phone’s camera with various additional information. When we use our phone to scan the sky, the names of planets, stars, and satellites above us pop up on the screen. When we scan the horizon, our screen will provide the names of individual mountains. And by taking a photo of the street we walk in, we’ll be able to read reviews of the local coffee shops, while direction arrows will guide our way to wherever we are headed.
The introduction of this new technology in our daily lives is an opportunity to question how our subjective image of reality is already being augmented by our brain while processing information forwarded to it by our senses. To which degree has our image of the everyday world already been complemented, processed or augmented?
Just like the modern cameras, humans are able to effortlessly recognize other people’s faces and keep eye contact during a conversation. If this system for automatic face tracking and recognition doesn’t function properly (which is often the case with autism-spectrum disorders), it can be difficult to make contact with our fellow humans. In this case, we need to put extra effort into distinguishing people’s faces from all the other images that surround us and to consciously focus our attention on them.
Seeing black-and-white in colour
Recognizing letters and symbols happens in a similarly automated way. Everyone who has learned to read can immediately understand the meaning of a written sign and react to it without any additional effort. We become conscious of our abilities only when we are faced with something written in a foreign alphabet. Those images on paper don’t make us automatically recall any meaning or emotion because our environment recognition system recognizes the text as a mere drawing and not as a combination of letters. On the other hand, people whose brains tend to confuse some of the sensual input they receive, usually have a very interesting experience of augmented reality. For instance, a girl telling her flabbergasted parents how pretty it is that the letter P changes its colour when she turns it into R. When the girl drew an additional line to the P, transforming it into R, she observed that the letter changed from yellow into orange, even though it was written in black ink on white paper.
The phenomenon of mixing sensory impressions, such as attributing different tastes or colours to letters, cyphers or images, is called synaesthesia. The most common experience of synaesthesia is recognizing a specific colour of cyphers or letters, while the less common is sensing a characteristic smell, taste or feel of sounds, images or people. Some people with synaesthesia therefore have a very direct experience of being touched by a piece of music or speech, because they actually feel physically touched by certain sounds.
Many people with synaesthesia don’t realize for a long time that their world is different from that of most of their friends and relatives and that their brain adds features to their image of the outside world that aren’t really there. They usually realize their perception of the world is different only when they openly discuss their experience with other people. In the past they tended to keep it to themselves because they couldn’t rationally explain it and were afraid other people would think they were crazy. Today, kids often tell their parents they like the purple colour of the letter A or the green they see when they listen to the weather reporter.
Physicist and Nobel Prize winner Richard Feynman saw all the numbers and equations in colour and sometimes wondered whether his students had a hard time following his derivations on the blackboard because they only saw them in black and white. Writers, painters and composers have also reported about their experiences with synaesthesia and how the mixing of sensory input has helped with their creativity.
Playing a trick on perception
Synaesthesia is a good example of how our sensory perception of the outside world is in no way similar to a photo or movie camera that merely transfers an image to film, but is an infinitely more complicated system based on recognizing objects from the environment and placing them in our internal model of the external reality. Our senses are nothing but tools that are constantly checking whether our brain’s image of the outside world matches their sensory input.
Similar to how people with synaesthesia add extra colours, tastes or smells to certain symbols they recognize, the completely ordinary image of our environment is also far more complex than some passive frame of the outside world. It is adjusted in a way that helps us deal with our daily challenges. The image of the world that our brain keeps updating is very much augmented as it is, and is based on our memories, experiences and the knowledge of our surroundings. Similar to the augmented reality on our smartphones, our own representation of reality includes additional information that helps us react to events faster and with less effort. These are generally subconscious processes, but we sometimes become consciously aware of these extra information through the emotions they trigger.
This trait of the human brain, that it is always analysing and updating our sensory experience, can also be easily deceived. We can notice an increasing number of objects in our surroundings that are made especially in order to trigger a desired emotional reaction. They want to trick the brain into perceiving something it already knows, even though the actual object has very little to do with what the brain takes it for. Decaf, artificial sweetener, non-alcoholic beer, and other similar products are designed to trigger the same response in us as the real thing, even though they are clearly lacking in ingredients.
With the rise of the augmented reality technology, some kind of decoy objects are bound to get invented eventually, with the purpose of tricking the smartphone algorithms into thinking they are sensing another object than the one registered by its camera or another smart sensor.
