With Emotion Recognition Algorithms, Computers Know What You’re Thinking

12,901 11 Loading


Back when Google was first getting started, there were plenty of skeptics who didn’t think a list of links could ever turn a profit. That was before advertising came along and gave Google a way to pay its bills — and then some, as it turned out. Thanks in part to that fortuitous accident, in today’s Internet market, advertising isn’t just an also-ran with new technologies: Marketers are bending innovation to their needs as startups chase prospective revenue streams.

A handful of companies are developing algorithms that can read the human emotions behind nuanced and fleeting facial expressions to maximize advertising and market research campaigns. Major corporations including Procter & Gamble, PepsiCo, Unilever, Nokia and eBay have already used the services.

Companies building the emotion-detecting algorithms include California-based Emotient, which released its product Facet this summer, Massachusetts-based Affectiva which will debut Affdex mobile software development kit in early 2014, and U.K.-based Realeyes, which has been moving into emotion-detection since launching in 2006 as provider of eye-movement user-experience research.

emotion-detection-aiThey’ve all developed the ability to identify emotions by taking massive data sets — videos of people reacting to content — and putting them through machine learning systems. (The startups have built on a system of coding facial expressions developed in the 1970s for humans to carry out.) Machine learning is the most straightforward approach to artificial intelligence, but it's mostly limited to deductive reasoning and isn't likely to give rise to nuanced artificial intelligence.

Yet, with the ability to capture, in video freeze-frame, fleeting expressions that are too quick for a human to definitively identify, the algorithms may already be smart enough to provide more information on what people are thinking than has ever before been available.

“The unguarded expressions that flit across our faces aren’t always the ones we want other people to readily identify. We rely to some extent on the transience of those facial expressions,” Ginger McCall, a lawyer and privacy advocate based in Washington, D.C., told the New York Times recently.

Here's how the systems work. Facet breaks facial expressions down into 44 distinct movements. Emotient, a University of California San Diego spinoff, says its algorithm can identify joy, sadness, anger, surprise, fear, disgust and contempt. It offers its service as a software development kit, or SDK, for others to use in their apps. For instance, a video game could pick up the pace if the player appears bored.

Affdex is a cloud-based software application that subscribers interact with using a dashboard that allows them to generate reports, export data and bookmark videos. It identifies both emotions and the intensity of the user's attention.

affdexAffdex originated with academic research founder Rana el Kaliouby conducted into autism. Those affected by autism can’t recognize emotion in human expressions the way most people can, and el Kaliouby first sought to use emotion-labeling computers to assist autistic patients. Marketers quickly saw the utility of the system for them and el Kaliouby spun off a company.

Realeyes relies on similar algorithms, but the company recruits viewers who fall within a particular target audience to vet the content. (See more coverage here.) As a result, it takes 48 hours to get results, whereas the other services are offered in real-time.

In addition to the advertising arms race to deliver the most “relevant” content to bleary-eyed web surfers, the presence of a webcam on nearly every laptop and mobile device has propelled this use of artificial intelligence. Not all of those webcams provide an image clear enough for the algorithms to decode, but the companies are betting on continued improvements.

We’ve all heard about webcams being hijacked — could this technology be used on unsuspecting users? That depends. It is ostensibly illegal to record an unwitting user via webcam, but that doesn't make it impossible.

"It would be very difficult for the companies to ensure consent of third parties, especially if the technology becomes mobile — that is, if it is used as part of a mobile application. We've already seen instances where similar technology has been used in televisions without consumer consent," privacy advocate McCall told Singularity Hub.

And if a mounted camera on private property did the recording, emotion detection would likely be legal. Job interviews may have just gotten a lot harder — just be careful not to frown.

Photos: Realeyes, Emotient, Affectiva

Cameron Scott

Cameron received degrees in Comparative Literature from Princeton and Cornell universities. He has worked at Mother Jones, SFGate and IDG News Service and been published in California Lawyer and SF Weekly. He lives, predictably, in SF.

Discussion — 11 Responses

  • Warwick January 19, 2014 on 4:53 pm

    Combine this with something like Google Glass and our social world would be in tatters. The capacity to know what others are really feeling would be too tempting for many but the results would likely be pretty disruptive for social interactions. I’m not one to be spooked by technological developments but this one, if made portable with live feedback, could have a lot of very unpleasant consequences.

    • didibus Warwick January 20, 2014 on 2:33 pm

      What kind of unpleasant behavior? Can’t you already tell someones mental state anyways? Say that an algorithm could actually be more accurate than our own instinct, I still don’t see what would be so socially transformative from it. Maybe you’ll know that your boss is actually let down by your work, or that your wife isn’t happy anymore, etc. I don’t see any bad with that. In fact, I think this could be very positive.

  • Telcomcorp January 19, 2014 on 9:26 pm

    This has a lot of legal and moral implications, for instance – what if you could use this as evidence for a discrimination or bias.

    What if your being generally and covertly discriminated against and harassed – if internal psycho pathologies (like paranoia,etc) are ruled out then evidence for and diagnostics of social or group pathologies could be processed.

    • didibus Telcomcorp January 20, 2014 on 2:30 pm

      Minority Report anyone?

      But there is a simple solution. Don’t allow to use this as incriminating proof. This isn’t unlike lie detectors. I see it treated the same way.

  • SethGrimes January 20, 2014 on 7:09 am

    Folks who are interested in this topic might check out a conference I organize, with speakers including Emotient research scientist Jacob Whitehill, speaking on Automatic Facial Expression Recognition for Emotion-Aware Computing, and Affectiva co-founder, Prof. Rosalind Picard of the MIT Media Lab, whose keynote is titled Adventures in Emotion Recognition. The conference is March 5-6, 2014 in New York; visit http://sentimentsymposium.com/ for information and to register.

    Seth Grimes, http://twitter.com/SethGrimes

  • didibus January 20, 2014 on 2:28 pm

    We’re already capable of understanding each other’s emotions. I feel this article is trying to be a bit fear inducing. I really don’t see the issue it could have that computers could also understand our emotions.

    • Luka Verigic didibus January 21, 2014 on 10:10 am

      ^ +1 The narrative suggests its a bad thing… I say its kickass 😀

    • Cameron Scott didibus January 29, 2014 on 11:35 pm

      As with all other technology, whether it’s good or bad depends on how it’s used. One researcher mentioned in the post has used the capabilities to help autistic children and to spur Web users to buy more stuff.

      • Matthew Cameron Scott February 2, 2014 on 2:22 pm

        great points all around 🙂 I am proud to see such an acute concern for avoiding the same type of negativity bias that we see far too often in mainstream media (as described in the book Abundance: the future is better than you think). it makes us too angry/scared/apathetic. and as we have discovered in ai research reverse engineering the human brain, arguments lead to no new relevant information. This movement is not about the amygdala, it’s about the frontal cortex, and mirror neurons, and empathy. it must be about advancing positive innovation to protect human rights and the environment. we have to adhere to the infinitely complex yet simple to understand patterns of life, aesthetics, ecology, and human rights. just like peter thiel said in singularity summit 2011, ‘as we progress into the future and our capabilities increase we need to be aware and have both parts increased optimism, and increased perssimism.’


        …I say mostly increased optimism to facilitate a more productive life to the cause. this reality is mostly good, and we’re getting better. as we know from the book abundance, and this video:


        …where Michael Shermer has articulated with several historical empirical statistical facts, that as we progress into the future there is an intertwined force transforming us into a better reality (like the force of electromagnetism/light)–capital (our material capabilities) and liberal democracies (our collective will/human rights).

        i am starting to get the feeling… that our human rights are advancing faster than science is able to help us… we now treat each other better than ever as a whole. remember shermer showed the charts about decreasing hate crimes and murders, racial homocides, etc. you can see it in the media. we have voted gay marriage in. we no longer have problems with ourselves… we are more unified than ever despite this alleged polarization. I feel there is not enough focus in the science community on political and economic and other types of science that are required specifically to advance human rights. as soon as we abolish the final remaining vestiges of fatal human tyranny–economic inequality–things will be better for everyone. including the super rich. capital is great as long as it circulates. which it is not. it’s getting syphoned up (not trickling down). people also said slavery would ruin our economy. we can take the increasingly marginal amounts of capital it would take to correct minimum wage AND create more jobs. it’s not either or as we’re told by classical economists languishing on this idea of limited resources. limited material resources are a thing of the past. since 2008, solor power has decreased in price by 80% … coal power has increased in cost by 15% … this is not our daddy’s economy. we do not need oil as we are told. just like we don’t need this asinine to own post industrial weapons of death to. what about the less conversed, more important, in total contradiction constitutional right to life? we no longer need to war over our limitations for some greater good. we can have our cake and eat it too. minimum wage should be corrected (not increased) to $21.72/hr. this economic circulation would stand to make the rich even more money, and would allievate a LOT of suffering for humanity. I am not saying abolish capitalism. it is overall a positive force as shermer showed us that was invented as a borometer of our craftsmanship/contributions to society that was necessitated in the past in a parallel world where agriculture led to an era of abundance and automation. there are a lot of good philanthropists/aristocrats in the world. we tend only to talk about the screw ups…

        I just get the feeling that science (like art used to be distorted by it’s patrons, the church) could be a little less invested in glowing bunnies and house plants, faster computers, more powerful weapons, or reading our minds, and making more money… and perhaps a little more invested in human rights and ecology which is an ecological, national security, and human rights and economic imperative–I know from the book “Science for Sale: The Perils, Rewards, and Delusions of Campus Capitalism.” that scientists are stripped of their personal endeavors and absorbed into corporations to do specific things for which they don’t even care, to make other people rich. kind of like how independent artists or mom and pop shops become disenfranchised in certain ways by major corporations. it is imperative that we find gainful employment for 7 billion intelligent beings… co exist peacefully, and advance activism and human rights through voting, (the things we buy is also like voting. it changes the material world just as much, for better or worse). let’s all invest like Kurzweil said in singularity, in exponentially growing fields like gnr (genetics nanotechnology robotics), computation, aerospace, 3d printing, biotech, software, etc. if they don’t repatriate the fractional sums of capital necessary to take care of we the working consumers who created them, then let’s occupy wall street for real and just take it back ourselves. shed light on and invest in corporations we believe in (do they advance human rights? ecology? fair trade with other countries), make each other rich, and accelerate a better world. money is the new voting. we all do it anyway every time we buy a phone, use energy, go to the doctor, click a movie on Netflix/amazon, etc. let’s take it back. it’s the beauty of the free market. we need to stop enslaving and subjegating each other and go even further to have inclusive empathy and help those in need. the ways in which we co exist will make or break us.

        • Matthew Matthew February 2, 2014 on 2:33 pm

          things are getting better on average all the time, but to those still suffering… they are not fractions. I just feel like there is still too much suffering/inequality for how capable we now are and future people will look back and wonder why it took us 150 years to share and do things right. that said I am more optimistic than ever and I am sure most of us will be bored at worst. there are no terrorists, 5 thousand people died that one time in 9/11. that’s 5 times less than die every single year from guns and poverty and health problems that should no logner exist. it would take 50 billion dollars to educate everyone in the world by 2015 yet we invest 80 billion in defense being afraid of the “terrorists.” we will figure this out…

          • Matthew Matthew February 2, 2014 on 2:57 pm

            and don’t let people pull the “you’re being political” card when you share some interesting facts. everything got political the second 7 billion people appeared on the planet. we will find a way to coexist with each other and the environment. life is good.