The Uncanny Valley describes a sense of unease and discomfort that some people experience when looking at a realistic virtual human. Its takes its name from a dip in a graph as the human likeness of a character increases .
Despite numerous books, papers, brain imaging studies and pop culture references (with a gold medal going to Frank from 30 Rock) we aren't even sure if the Uncanny Valley actually exists. It is entirely possible that we simply haven't built a convincing enough virtual human yet.
In this post i'd like to return to a suggestion I made 11 years ago that the Uncanny Valley may be due to a clash between visual cues indicating believability and behavioral cues indicating falsehood. In other words, if you build a crazily high fidelity face like Paul Debevec's model of Barrack Obama above, you need to match this with an equally high fidelity animation. If you don't 100% nail the alignment between visual realism & behavioral realism then the mismatch is subconsciously jarring.
Fear of the Uncanny Valley has led to many character designers dialling back visual realism in favour of more abstract designs that don't raise an expectation for high levels of behavioral realism. However, abstract avatars do not completely remove the problem of clashes between mismatched graphical and behavioral cues. For example, when a network connection freezes and an avatar is left staring into space, this is perceived as coldness and a personal affront rather than a technical glitch. This is because incorrect body language in an immersive environment is automatically interpreted as a social signal, rather than a technical error.
To make things even more challenging, the illusion that you are sharing a space with a living being quickly breaks if the character fails to respond correctly to a user's body language and behavior. To illustrate this point, consider this moment in the immersive documentary Clouds over Sidra when a group of children in a refugee camp run towards you and crowd around your feet.
You look down and a young girl seems to look up at you and make eye contact, creating the illusion of a sudden, intense communication with another human. This moment of connection is quite unlike seeing a character in non-immersive media because you feel that you are sharing a space with someone whose gaze is focused directly on you, in 3D space. Eye contact within this pre-rendered footage is powerful enough to create a strong appeal to empathy. However, the momentary illusion of life is quickly broken because the girl does not respond any further to your body language. Adding this type of moment-by-moment responsiveness is the missing ingredient for realism in virtual reality.
If you want people to respond to VR as if its real. VR should respond to them as if they are real.
Creating responsive virtual humans is a complex problem that will require sophisticated methods to continuously sense and interpret a users behavior. But solving this challenge will be well worth the reward.
An animated character looking at me that's one thing. A human being looking right at my eyes, interacting with me that is not there with me. That's a man on the moon moment. That's the first telephone call.