When Your Computer Watches Back

I frequently joke about journalists who use the term “Orwellian” to describe neuromarketing, but Orwell’s novel 1984 did foresee one technology that may become a reality: a television (or at least a monitor) that watches you back. The technology to have a webcam observe your facial expressions now exists, and may be deployed in ad platforms, games, and other areas.

For years, a small group of academics and marketers have used “facial coding” techniques to analyze the emotional reactions of people to stimuli. Facial coding experts believe that humans are incapable of completely hiding their true emotions. Even if you put a smile on your face when you meet someone you dislike, practitioners of this specialized art believe that your true feelings (anger or disgust, for example) will flicker across your face before you plaster on your “social smile.” The field was pioneered by academic psychologist Paul Ekman, and facial coding remains the primary tool for market research firm Sensory Logic. Law enforcement agencies have occasionally employed facial coding analysis as a sort of lie-detector when questioning suspects and witnesses.

Realeyes

The tricky thing about facial coding is the need to have a trained expert study the expressions in real time or on video. This reduces scalability and also introduces the possibility of variation in interpretation between different viewers. One company, UK-based Realeyes, claims to have automated the process of gauging emotional reactions. By combining emotion-monitoring and conventional eye-tracking, the firm says it can accurately measure viewer engagement with web ads, web sites, print ads, etc. Currently, the work is conducted in a network of collection centers across the US and Europe, and sample size is reasonable: 50 and up is typical, according to their website.

Now, Realeyes says they can greatly expand their reach by using webcam based facial analysis, according to The Economist:

One of the companies doing such work, Realeyes, which is based in London, has been developing a system that combines eye-spying webcams with emotional analysis. Mihkel Jäätma, who founded the company in 2007, says that his system is able to gauge a person’s mood by plotting the position of facial features, such as eyebrows, mouth and nostrils, and employing clever algorithms to interpret changes in their alignment—as when eyebrows are raised in surprise, say. Add eye-movement tracking, hinting at which display ads were overlooked and which were studied for any period of time, and the approach offers precisely the sort of quantitative data brand managers yearn for.

Affectiva

Another firm mining similar territory is US-based Affectiva. Their Affdex system works over the web via a webcam, and identifies when a viewer smiles, frowns, etc.

Affectiva also combines remote biometric monitoring with their eye tracking and facial monitoring. This video explains some of the science behind their techniques:

Microexpressions?

One thing that’s not clear is how well these web-based solutions can capture the fleeting microexpressions touted by human facial coding experts. These must often be captured on video and viewed in slow motion. Even if the current webcam technologies don’t allow it, it seems inevitable that resolution, frame rates, and software will eventually catch up with lab-based cameras.

While initial applications may be geared to analyzing ads and other content, the real potential of these tools could be realized if they allowed ads, games, and websites to react instantly to changes in viewer emotion. Content that produced a positive response could be extended, while content causing an unexpected negative reaction could be replaced by other content.

It’s almost Orwellian. 😉