Turning Data Into Knowledge

Connecting the physiological with the cognitive to predict behavior. JL

Bill Cannon reports in Scientific American:

Conducting experiments to identify physiological correlates of cognitive overload, “we attach a lot of sensors to an individual to collect brain and body signals." They monitor pupil diameter, eye activity, respiration, and voice stress, all while putting participants through scenarios. “We process signals with algorithms to extract features from them. We have found that there are certain physiological features associated with mental overload, and they’re pretty consistent.”

In a YouTube video, a drone pilot sits at a pair of monitors, observing convoys that off-camera radio voices describe alternatively as “friendly” and “suspicious.” Dramatic movie-trailer-style music builds as new radio and visual information comes in fast and furious. A close-up reveals the pilot’s sweat-drenched face; his eyes dart wildly. The music crescendos as the video locks in on the pilot’s panicked eyes.

The pilot, a narrator explains, was presented with jigsaw-puzzle pieces of information—weather, flight data, intelligence about friendlies and enemies—with important data buried or missing. He started by following two “remotely piloted aircraft,” the military’s preferred term for drones, then a third and a fourth. This led to a “loss of situational awareness,” the video’s narrator explains. “The pilot entered the state of information overload and started falling behind. The mission performance began to suffer.” Fear, confusion and frustration impaired his judgment and his brain went into fight-or-flight mode, pumping out adrenaline and other stress hormones that raise blood pressure and significantly increase heart rate, which reduces a person’s ability to perform. If the mental load is more than brain and body can handle in such a military situation, the outcome could be lethal.

The video—produced by the Air Force Research Laboratory’s 711th Human Performance Wing’s HUMAN (Human Universal Measurement and Assessment Network) Laboratory at Wright-Patterson Air Force Base in Ohio—shows how researchers assess pilots’ struggles to juggle information and measure stress as they become mentally overloaded and mistake-prone. 

Assessing Overload

The HUMAN Laboratory’s manager, Matt Middendorf, says that over the past three years, his team has been conducting experiments to identify physiological correlates of cognitive overload. “We attach a lot of sensors to an individual to collect a wide range of brain and body signals, including an electroencephalogram [EEG]—that’s brain activity—and an electrocardiogram for heart activity,” he says. They also monitor pupil diameter, eye activity, respiration, and voice stress, all while putting participants through scenarios like the one in the video. “We process those signals with algorithms to extract features from them,” Middendorf explains. “We have found that there are certain physiological features associated with mental overload, and they’re pretty consistent.”

Perhaps the surest and simplest sign is heart rate. “It goes up when you’re stressed out,” he says. At the same time, “heart rate variability goes down.” That is, the inter-beat interval becomes more uniform as stress increases. More variability in heart rate is good—as long as it is not too variable—and too little variability is bad for a person’s overall health.

Another key assessment metric uncovered by the work in the HUMAN Lab lies in the eyes: Pupils dilate under stress but, Middendorf cautions, researchers must factor in the room’s ambient light levels, which influence pupil diameter. Also telling of cognitive overload are blink duration—how long your eyes are closed—and blink rate. Under high stress, blinks become shorter and less frequent.

Evaluating Interest

At Lockheed Martin’s Advanced Technology Laboratories (ATL) in Cherry Hill, New Jersey, William Casebeer—manager of human systems optimization in ATL’s human systems and autonomy research area—and colleagues are building “generation-after-next science and technology” for other Lockheed business units and the U.S. Department of Defense that “allow humans and their autonomous machine teammates to work together more effectively,” says Casebeer.

To do that, Casebeer and his team, like their Air Force colleagues, have been gathering data while putting subjects through various tasks. He offers an example. “If you are an intelligence analyst who’s looking at photographs, and you’re trying to identify objects of interest in them, it might be useful to monitor EEG data for signals,” such as the P300—a positive energy wave that can be measured over the scalp’s surface in the first 300 milliseconds after the presentation of a new image. “And then the assessment algorithm will tell you that, ‘Oh, that signal was the strongest for these particular photographs, so spend more time looking at them because it’s more likely there’s an object of interest,’” Casebeer says. “So the assessment piece of the puzzle is just making sense of that sense-data in the context of task.”

ATL colleague and senior scientist Bart Russell studies another assessment challenge: oscillations between busy time and its opposite, the lull. “The hurry-up-and-wait phenomenon,” she calls it. “Our eyes don’t always reveal what our brain is doing, whether or not we’re engaged in the situation.” There are different levels of engagement, as well as misdirection—as she and others in the field put it, the eyeball versus the “mindball.”  We’ve all had the experience, for instance, of looking straight ahead but with our attention actually on something in our peripheral vision. Russell’s group is “working on a method that actually can sense engagement” that combines infrared eye-tracking techniques with EEG to determine attention lapses and what “information a person is attuned to even if they’re not looking directly at it.”

As their work at ATL progresses, they’d like to work in other assessment metrics like low blood glucose levels, which can affect cognitive performance. “We don’t have a good sense about how to integrate blood glucose levels with electrophysiological technologies,” Russell says. “But as healthcare companies develop wearable glucose monitors, we can start to integrate those signals into a broader framework.”

Exercising New Options

The broader framework has been the terrain of Northeastern University’s senior vice provost for research and graduate education, Arthur Kramer, for the better part of three decades. Working with hundreds of test subjects, he has studied the structure and function of the aging brain and shown that physical activity improves cognition and brain health.

In recent years, Kramer has expanded his work to include younger groups and has found similar results across the age spectrum: “Exercise can improve various aspects of cognition—including memory, decision-making, attention,” he says. “In addition to that, it can improve or increase brain structure, or brain volume, in areas that show age-related decline,” such as the hippocampus. 

As Lockheed’s Russell predicts, variety will likely be key for the next generation of assessment techniques, which she hopes can tease out individual differences in mental overload, attention lapses and cognitive decline. “The military is a one-size-fits-all organization,” she says. “As we move forward with these tools, it’s not just desirable to have them individualized—it’s a requirement.” 

Turning this military research into individualized solutions will also benefit the healthcare community, and researchers at The Ohio State University are coordinating the clinical uses that will improve the lives of patients with diseases and injuries.