I frequently joke about journalists who use the term “Orwellian” to describe neuromarketing, but Orwell’s novel 1984 did foresee one technology that may become a reality: a television (or at least a monitor) that watches you back. The technology to have a webcam observe your facial expressions now exists, and may be deployed in ad platforms, games, and other areas.
For years, a small group of academics and marketers have used “facial coding” techniques to analyze the emotional reactions of people to stimuli. Facial coding experts believe that humans are incapable of completely hiding their true emotions. Even if you put a smile on your face when you meet someone you dislike, practitioners of this specialized art believe that your true feelings (anger or disgust, for example) will flicker across your face before you plaster on your “social smile.” The field was pioneered by academic psychologist Paul Ekman, and facial coding remains the primary tool for market research firm Sensory Logic. Law enforcement agencies have occasionally employed facial coding analysis as a sort of lie-detector when questioning suspects and witnesses.
Realeyes
Now, Realeyes says they can greatly expand their reach by using webcam based facial analysis, according to The Economist:
One of the companies doing such work, Realeyes, which is based in London, has been developing a system that combines eye-spying webcams with emotional analysis. Mihkel Jäätma, who founded the company in 2007, says that his system is able to gauge a person’s mood by plotting the position of facial features, such as eyebrows, mouth and nostrils, and employing clever algorithms to interpret changes in their alignment—as when eyebrows are raised in surprise, say. Add eye-movement tracking, hinting at which display ads were overlooked and which were studied for any period of time, and the approach offers precisely the sort of quantitative data brand managers yearn for.
Affectiva
Affectiva also combines remote biometric monitoring with their eye tracking and facial monitoring. This video explains some of the science behind their techniques:
Microexpressions?
One thing that’s not clear is how well these web-based solutions can capture the fleeting microexpressions touted by human facial coding experts. These must often be captured on video and viewed in slow motion. Even if the current webcam technologies don’t allow it, it seems inevitable that resolution, frame rates, and software will eventually catch up with lab-based cameras.
While initial applications may be geared to analyzing ads and other content, the real potential of these tools could be realized if they allowed ads, games, and websites to react instantly to changes in viewer emotion. Content that produced a positive response could be extended, while content causing an unexpected negative reaction could be replaced by other content.
It’s almost Orwellian. 😉
Human beings are experts in facial reading almost from birth — our survival depends on our ability to read the social environment: other people and their responses. The fact that this skill can be codified and “precised” just extends that natural ability to do the math on it. The real issue is what’s happening in the brain: WHY these responses are occuring, which is the black box problem that marketers keep stumbling on. Even if we can read reactions, we still need to track those back to their causes. Cultural reasons, and universal biological psychology, is the territory where answers have to be sought out, identified, and described.
I agree, Margaret, though most of us may think “something’s off” or “he seemed friendly, but it seemed kind of phony” rather than actually noting what expressions or behaviors made us feel that way. Some companies like Sensory Logic say they can interpret the expressions, including fleeting ones, in a way that helps them improve ads.
Roger
I find this research interesting. I do have to keep reminding myself that Paul Ekman cautioned that these are indicators only. Given enough time or if the fabricator is well practiced, then the fabricator can still be deceptive. Nevertheless, his research seems to be a sound foundation for continuing efforts to explore what messages we are sending and receiving. What made him a great reseacher to me was his skeptic attitude. He was careful about conclusions. As marketers, we must be careful about our conclusions. For example, a head tilt by a young female can have different meanings. There are few silver bullets in a marketer’s revolver.
Good points, Chester. One good indicator that expressions can be fabricated is the movie industry. Most actors are quite convincing at simulating a range of emotions. I’m sure many non-professionals can do so as well.
Good scientists are always skeptical, I think. The moment you fall in love with your theory, you start down the road toward ignoring data points that don’t fit as “experimental error” and taking other liberties.
Roger
facial coding looks really cool thing. I never heard about it before but I’m happy to hear about it now.
I’m not too sure I want my monitor looking back at me, especially while I’m working in my pajamas. 🙂 The technology is fascinating though and I’m sure marketers would love to have their hands on that kind of information. The public though I think would have a different opinion.
Rusty, I expect we’ll be able to enable this technology only when we want to, unlike Big Brother’s all-seeing eye.
But, what if merchants found the technology really useful and effective – would you opt for an free Kindle Blaze 3D, free with ads and continuous facial monitoring?
Roger
The trouble with invasive technology is that people get immune to it. If someone told the world when the internet started that they’d have their name, photo, birthday, relationship status and all their interests (not to mention that one drunken night you’d rather not remember) public for everyone to see there would have been an outcry. Today’s social internet has changed the face of privacy and I think many kids and adults aren’t aware of the ramifications.
By the way you can send my free Kindle Blaze 3D to 7625 Waverly Way … 🙂
Fascinating stuff, Roger, thanks.
It brings to mind the famous Turing Test — what’s the difference between a machine that can sense your facial gestures as well as a human, and a human itself.
Well, on re-reading this article about the Turing test, I’m no longer troubled by that question. The real issues are whether or not I care about someone collecting that sort of data, and for me, that’s not too hard either; if I’m walking about in the world, I’m offering that sort of data about myself to anyone who might be watching.
Rusty’s right that this sort of technology has changed the world, and we are all leaving little droppings of info about us all over the place, ready to be skimmed up by a plethora of clever technologies. But then, I find myself less and less bothered by it.
I forgot to cite the Turing test article; here it is.
http://www.thenewatlantis.com/publications/the-trouble-with-the-turing-test
Charles, I agree that we are all getting used to more info about us being collected and, at times, exposed. A few years ago, who would have thought that you could look up someone’s address and see detailed aerial and street view photos of it?
You are the trust guy – do you think we tolerate this data collection because we trust the people and companies gathering it not to abuse it? Or have we just thrown in the towel?
I’veheard bits and pieces about facial coding and was always intrigued by the theory and science behind it. I’m not too sure I’d really want my PC to look back and analyze me, but hey…. look at all the current technology we have now. It’s all about moving with the times and getting used to the trends I guess.