How Do You Feel? Your Computer May Know

Our ability to measure emotion and chart its impact continues to grow. This space has posted recent articles on how that is being used in advertising and other forms of communications. Now we learn that researchers may be able to enable devices to detect moods. The implications are, as always, fascinating and somewhat Orwellian. The trade-off between enhanced convenience or safety with the potential for abuse will continue to be debated but the bet here is that nothing will stop this type of analysis. The following is an interview between NPR host Dr. Ira Flatow and MIT Media Lab researcher Rosalind Picard:

"Researchers are developing devices that measure and track emotions, while subjects are online, driving or even shopping. By measuring emotional responses, scientists say they can improve safety, communication, performance or even help marketers come up with better products.

IRA FLATOW, host:

What if our gadgets were, well, a little more sensitive? What if they knew that we were getting frustrated or mad or even tired and they could tailor our experience? Whether it's shopping online or driving our cars, they could tailor those experiences to suit our mood, maybe make us a little happier.

Or if your car knew you were - knew your driving habits so well that it could sense when you were getting angry, and it could intervene, sort of like a built-in road-rage-o-meter, and keep you from doing something stupid on the highway.

Well, scientists at MIT are working on devices, including web cams and special wristbands, that help measure emotions in people. The data they collect might help us make better communicators or drivers, or maybe the devices could one day help people with emotional or physical problems, like autism, communicate their feelings, communicate their feelings and needs better to us.

What do you think? Would you want your computer or a wristband to track your emotions and learn about them, and maybe interact better with you or maybe take care of you? Or do you think that's an invasion of your privacy?

Let me introduce my guest. Rosalind Picard is director of affective computing research and director of autism and communication technology at the MIT Media Lab. She's also professor of media arts and sciences at MIT and co-founder and chief scientist at Affectiva. That company is Affectiva, Inc. Welcome back to SCIENCE FRIDAY, Dr. Picard.

Dr. ROSALIND. PICARD (Massachusetts Institute of Technology): Thank you, it's a pleasure to be here with you.

FLATOW: Can you describe exactly what emotion sensing is and how our emotions are sensed?

Dr. PICARD: Well, first of all, what we're sensing is some outwardly measurable aspects of emotion, which is not quite the same as knowing what you're really feeling inside.

For example, we can, using a camera, recognize your facial expressions, and if you choose to show that you're frustrated or angry or smiling and pleased, then the camera can start to interpret that information and hopefully respond respectfully like a person hopefully would.

We can also put a sensor on - if you choose, it's all opt-in - and from the surface of your skin pick up some changes that are happening inside your body. And those can be indicative of some aspects of your emotions changing as well.

FLATOW: What kind of changes would it sense?

Dr. PICARD: Well, the main one that our sensor picks up on now is the electro-dermal activity. That's small changes in conductants sensed from the surface of your skin. And those are activated when your sympathetic nervous system, sometimes known as a fight-or-flight system in your brain, is activated.

So when that fires, it affects the innervation of the nerves around the sweat glands under the skin, and even before you sweat, we can pick up those changes.

FLATOW: How different or similar is this to what is being sensed on polygraph tests?

Dr. PICARD: This is one of the signals used in polygraph tests. But it doesn't just change with lying. For example, it can go up with things that you're positively excited about, like getting ready to meet somebody very special to you. It can go up.

FLATOW: And you work with these - with the idea that we might be able to get our machines, which they all have computer chips in them right now, of course, to be able to interact with us better and understand our emotions or things that we're feeling, and you're also looking at it for possibly helping out kids with autism.

Yeah, there are a lot of people for whom our technology is actually useful right now. Well, you know, the general emotional intelligence of machines still kind of has a ways to go. But right now we have people who are wearing our sensors, say, who are unable to speak and unable to talk about their feelings, and people are kind of trying to guess what's going on by looking at them on the outside, and that's extremely inaccurate.

And while the sensor is not a perfect readout of your emotions, it does tell you some changes that you can't always see accurately on the outside, and those can be really valuable.

FLATOW: When you take these measurements, how many do you - you know, how do you calibrate the machine? What's the process that goes on here?

Dr. PICARD: We actually don't have to do any lengthy calibration. You can just put the sensor on and see the level and then watch how it changes within a person. So when a person's relaxed, it goes down. When they get excited about something, it goes up.

So it's pretty easy. Now, people may have different absolute levels, but it's very easy to adjust for those by just looking at them over the course of the day.

FLATOW: Is it easier to tell when someone's getting angry or getting happy?

Dr. PICARD: Interesting. Usually we would look at the face for that. If you're getting really happy, both could be activating for our sensor, anger and happiness. And hopefully the face would tell you if it was positive or negative.

However, we've just presented some results at a conference I'm at in California, showing that sometimes when people are frustrated, they respond with a happy smile, and that may be a coping mechanism.

So it's important to look not just at the smile and not just at the autonomic response.

FLATOW: I've seen commercials recently for automobiles that will detect when you're falling asleep or avoid oncoming objects if you're a little drowsy. Is this that same kind of idea here, at least in that frame reference?

Dr. PICARD: We haven't focused on driver drowsiness, but the same kinds of signals could be used for that, yes. So you could look at things like: Are the eyes closing? Is the head falling forward? Is the person's muscle tone kind of fading? And is the skin response, which also shows very interesting patterns during sleep, changing in a way that looks like the person's falling asleep?

FLATOW: Do you find that people are eager to use this, or do they see this perhaps as an invasion of their privacy?

Dr. PICARD: Well, I mean, if somebody forces this on you, I would see it, you know, as an invasion of privacy. But our technology is all designed to be opt-in, and you can you can wear it or not. You can turn your camera on or not.

So I would certainly not force this on anybody. I think what we find is a lot of people are very curious about what's going on inside themselves, and they want to try it, and when they see it, they learn things, and that's really helpful to them.

So, you know, if you want to learn about yourself, then, you know, give it a try. You can keep the data private, or you can share it with others if you feel you trust them.

FLATOW: And if we wanted to try this, how could we do that?

Dr. PICARD: Well, the - I'm delighted to say the sensors we've invented at MIT no longer have to be sent out with a grad student and a soldering iron attached.

Dr. PICARD: Affectiva is selling them. If you go to Affectiva.com, you can learn more about them, or mail info@affectiva.com. They are providing the sensors primarily right now for researchers, and we are planning to come out with a consumer version later in the year.

MARY (Caller): Hi, thanks for taking my call. I actually like to sing in my car when I'm driving. And, you know, like if it's something that I really enjoy, I might get into it. And I'm wondering if someone were to be singing, like, an angry song or an emotional song, if the software that you're talking about, you know, interpreting our driving and changing the way our car responds, if that might have a negative, you know, impact or reaction if we're, you know, enjoying the music.

Dr. PICARD: I certainly think we should be very wary of how we have the car respond differently to us in our affective state. Some of the things people have done that I thought were really smart were, for example, if your GPS is talking to you and you're angry, then it had better not sound cheery, right? The voice might want to change its tone to be more subdued.

On the other hand, if you're, like, really happy, singing, you know, cheery, and the voice reflects that, then it's been shown people actually drive more safely when that voice responds, becoming more cheery when they're cheery or becoming more subdued when they're angry. So that I would see as something that would, you know, improve our performance behind the wheel, and that - that I would welcome.

FLATOW: I'm sure it's disco music. Let me ask you a question, Dr. Picard. If your car - let's take this car example, because I think it's fascinating. If your car discovers that you're angry, might it speak up, like a friend would, say, and warn you to come down a bit, or you know, you might be on the brink of road rage here, soothing, something like that?

Dr. PICARD: Yes, it's very interesting. I think if there's going to be a interaction with personality, if it did that, and you didn't like it, it had better not keep doing that because that could make you angrier, and that would make you a less safe driver.

I've been told by others that anger equals three beers behind the wheel of a car. If you're really angry, don't hop in the car. Do something else for a while. And whatever it takes to get you to calm down would be what we would want to learn how to do to support you if that happened in the car.

FLATOW: So you might actually, though, opt in to choose something like that to warn you?

Dr. PICARD: Yeah, yeah, if you have a problem - like we're working with people who struggle with emotion disregulation, whether it's anxiety attacks or panic attacks or cravings that proceed substance abuse. And if we can measure that pattern unique to them and understand how to support them at that moment, then we can, through the mobile device, actually bring them a response to their sensed data that is hopefully helpful for them.

FLATOW: So I imagine this would be useful in a shopping situation, where...

Dr. PICARD: There are a lot of companies interested in shopping. I know - now that we're completely mobile and you can just wear this in regular life, you know, you go in the store and you can see, you know, what makes your signal go up. And usually it's bad stuff. Usually it's something like, you know, one of those frustrating check-out machines that's not, you know, not making sense.

FLATOW: So it detects that you're sitting there frustrated on line because the person ahead has 15 items in the 10-item line, and maybe it knows that and it offers you a coupon.

Dr. PICARD: Yeah, actually, we've been looking at ways that, you know, people might be - well, some people just intrinsically want to learn about their data. Other people might be, you know, a little reluctant. Like why would I turn on my camera on the Web, you know, when I'm shopping?

But if it's just like them looking at you in the store, and they give you a coupon, you know, at checkout, why not turn on your camera and let them see that you're confused by this screen that they're showing you at the check-out section?

FLATOW: Yeah, or - if you have one of those electronic carts now that have all these TV screens on them.

Dr. PICARD: Uh-huh. Yeah.

FLATOW: Maybe it could detect and then offer you something on that screen or point you to a sale that's going on in the store or something like that.

Dr. PICARD: I'm sorry, what do we do with sales?

FLATOW: Or it could point you to something on sale to make you a little happier, offer you something.

Dr. PICARD: Sure, yeah. You know, we can easily detect smiles of delight, people walking in and out and looking at things. And certainly if you give people an unexpected surprise that's delightful, they can feel really good about that experience and want to come back and even want to tell others about it.