Patent No. 5447166 Neurocognitive adaptive computer interface method and system based on on-line measurement of the user's mental effort

 

Patent No. 5447166

Neurocognitive adaptive computer interface method and system based on on-line measurement of the user's mental effort (Gevins, Sep 5, 1995)

Abstract

A human-computer interface uses neuroelectric signals recorded from the user's scalp i.e. electroencephalograms (EEGs) to alter the program being run by the computer, for example to present less or more difficult material to the user, depending on the user's neurocognitive on-line workload score. Each user is tested with a standard battery of tasks, while wearing an EEG hat, to calibrate a neurocognitive workload function. The calibrated function is user-specific and is obtained by modifying a neural network pattern analyzer which has been previously trained to index neurocognitive workload using data from a group of subjects performing the same battery of tasks.

Notes:

Government Interests

This invention was made with Government Support under contract number F49620-92-C-0013 awarded by the Air Force Office of Scientific Research. The Government has certain rights in this invention.


Parent Case Text

This application is a continuation-in-part application based in part on application Ser. No. 07/766,826 for "Non-Invasive Human Neurocognitive Performance Testing Method and System", filed Sep. 26, 1991 now U.S. Pat. No. 5,295,491, issued Mar. 22, 1994.

 

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a human-computer interface using neuroelectric signals recorded from the user's scalp, i.e., electroencephalograms (EEGs).

2. Description of Related Art

Conventional computer user interfaces are driven solely by behavioral responses (i.e. muscle activity) by the user with a keyboard, mouse, joystick, touchscreen, pen or similar device, or a data glove in a virtual reality system. In this type of system, the user receives information from the computer screen and audio system in the form of visual and auditory stimuli, (or haptic stimuli in a virtual reality system), processes them, and makes a deliberate behavioral action which the computer interprets and acts upon. Computers that interpret human speech, or that use devices to determine where the user is looking, are other examples of interfaces that are controlled by behavioral responses from the user. Another type of user interface, which is still in the experimental stage, accepts a user's "thought commands," as measured by EEG signals, to control the movement of a cursor on a display screen.

All these interfaces are intended to allow the user to operate and control the computer system. The computer system has no information about the amounts and types of the user's mental capacities currently being utilizied, or even about the user's state of alertness. This results in a situation in which the overall efficiency of the human-computer system is less than it might be. For example if the user is mentally overloaded, or at the other extreme, if the user is drowsy, the overall performance of the human-computer system will be limited by the ability of the user to process and respond to the information presented by the computer.

Advances in technology have resulted in more complex computer based or computer controlled systems which can overwhelm the user's ability to process and respond to the information presented. Examples of this include jet fighter planes, air traffic control systems, powerplant and factory control systems, emergency management systems, multi-window displays of complex relations in a large data base, securities trading systems, and video games which increase task difficulty beyond a user's ability. New multimedia and virtual reality technologies are likewise expected to produce situations in which a user is mentally overloaded. At the other end of the mental effort continuum, highly automated computer controlled systems can require so little input from the user that the user can become inattentive or drowsy, for example piloting a commercial airliner. Other situations which can cause boredom and resultant inattention or drowsiness include long duration instrument monitoring tasks such as watching radar or sonar displays for unusual activity.

The lack of on-line knowledge of whether the user is mentally underloaded or overloaded is also a major limitation in computer-aided instruction systems. The use of computer-aided instruction is greatly increasing because of its ability to present material at a pace directed by the user, as compared with traditional instruction in which everyone in the class receives the same material at the same rate. However, unlike a human teacher, a computer-aided instruction system has no way of knowing about the user's mental state and therefore can not optimally adapt the material to her or his needs. For example, when a user answers a question incorrectly during a computerized training program, the system does not know whether the user was not paying attention or whether she or he was trying hard and simply did not know or understand the material. In the former case, an alerting signal could be presented and then the same material could be repeated. In the latter case, it would be useful to know whether or not the user was employing an appropriate strategy to solve the problem. If so, a more detailed explanation of the material which was not understood could be presented. If the user was using the wrong strategy for the problem, an explanation of how to go about solving the problem could be presented. For example, it might be determined that at the time the user made an error when answering a question requiring visualization of how the parts of an engine fit together, he was using 75% of his cognitive capacity; visuospatial systems were at 45% of capacity, while verbal encoding and output systems were at 85%. From this information, the system could conclude that the user was trying to solve the problem with a verbal strategy which was not efficient for the problem at hand and could present the user with information showing him or her how to use a visuospatial strategy to solve the problem. There currently is no way to obtain this information except indirectly by querying the user about his or her mental state. Besides distracting from the flow of the instruction session, this approach can be inaccurate since people are not always aware of their mental state.

 

OBJECTIVES AND FEATURES OF THE INVENTION

Supplemental information provided to the computer about the user's neurocognitive workload could serve to optimize overall human-computer system performance by allowing the computer to adapt the type and quantity of information being presented to match the current mental capacity of the user. In this context, the term "neurocognitive" refers to those mental functions for which physiological indices can be measured. Similarly, the lack of on-line knowledge of whether the user is mentally underloaded or overloaded also limits the capability of computer-aided instruction systems. If the computer could tell that the user was mentally overloaded, it could slow down the presentation of material (less difficult level); or it could speed it up (more difficult level) when the user was underloaded.

What is required in all these situations is a means for the computer system to assess the overall level of the user's mental effort, as well as degree of utilization of major regional brain systems critically involved with perception, cognition, or action, and to use this information to adjust the presentation of information to the user to achieve an optimal level of mental workload. We call this type of computer interface a "Sympathetic Neurocognitive Adaptive Computer Interface". Such an interface would have great utility in computer-aided instruction applications, including multi-media and virtual reality training systems, where knowing the degree and type of mental effort of the user would facilitate the acquisition of new knowledge and skills by allowing the computer system to rapidly adapt the information presented to match the mental capacity of the user. Similarly, in situations where the user is being presented with information but is not required to respond, knowledge about the user's mental workload could be used to adjust the presentation of information. Such an interface could also be used in complex computer controlled systems, such as commercial transportantion and military systems, where the system could take over critical funcions from the user if he/she becomes overloaded or drowsy.

It is an objective of the present invention to provide a method and system for measuring the amount of a person's mental efforts and the degree of utilization of major regional brain systems using neuroelectric (EEG) with or without other physiological signals (e.g., eye, scalp or facial muscle and heart activity) in order to provide the computer system with information about the user's mental workload. These regional brain systems involved with perception, action, and cognition include, but are not limited to: the planum temporale, superior temporal gyrus, Heschl's gyrus and associated structures involved with auditory processing and speech perception; the occipital and inferotemporal cortices and the parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition; the precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with the planning, initiation, and execution of motor movements; the dominant hemisphere frontal operculum, dorsalateral frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions; and the networks of structures encompassing the prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention, in learning, and in working memory.

It is a further objective of the present invention to obtain the metric of mental effort noninvasively and on-line while the user is actively interacting with the computer system or while the user is passively receiving information from the computer.

It is a further objective of the present invention to measure the amount of the user's mental capacity being utilized and the degree of utilization of major regional brain systems so that the computer system can adjust the information it presents to better match the mental workload of the user.

It is still a further objective of the present invention to obtain a user-specific calibration for the metric of mental effort and regional brain systems utilization non-invasively using a combination of stimuli, behavioral tasks, physiological measures and neuroelectric signals to train the computer to recognize the different levels of mental effort and utilization of major regional brain systems for each individual user. The calibration process provides the basis for deriving a user-specific mathematical function for relating patterns of neuroelectric activity, with or without other physiological measures, to overall mental workload level, and to the degree of utilization of specific brain systems involved with a given task or behavior.

It is a feature of the present invention to provide a computer user interface which is sympathetic and responsive to the user's level and type of mental efforts by adapting it's operating parameters (e.g., form, content, speed, etc.) to optimally load the mental capacity and regional brain systems of the user. For example, the computer can speed up or slow down the rate of presentation of a multimedia training session if it determines the user's mental effort is not within an optimal range by comparing the user's current mental effort with the function derived during the calibration process. The computer could further change the distribution of information between visual and auditory modalities, or, for the visual modality, between linguistic and graphical types of information display, to take into consideration the degree of utilization of a user's visual and auditory, or linguistic and graphical brain systems.

It is a further feature of the present invention for the system to present information at the instant at which the user's preparatory attention is optimal as determined by measuring the user's mental effort and regional brain utilization prior to presentation of information and presenting the information only when level of mental effort and regional brain utilization are at appropriate values. Alternatively, the system can change the sense modality or form of the information or alert the user prior to presenting the information if the user's preparatory attention is not optimal as determined by measuring the user's mental effort and regional brain utilization prior to presentation of information.

Other features of the present invention include the ongoing measurement of the user's EEG to determine the degree of utilization of various regional brain areas, and measurement of other physiological signals of scalp and facial muscle, heart and eye activity, using two or more electrodes placed on the body surface. Yet another feature of the present invention is the processing of the neuroelectric and other physiological signals in time intervals ranging from 100 milliseconds to several hours to derive a metric of mental effort which is transmitted to the same or a different computer in order to adjust the operation of the computer or simply to record the user's responses.

SUMMARY OF THE INVENTION

In accordance with the present invention, there is provided a novel method and system for measuring the amount of a person's mental efforts and the degree of utilization of major regional brain systems in order to provide the computer with on-line information about the user's mental workload. These major regional brain systems involved with perception, action, and cognition include, but are not limited to: the planum temporale, superior temporal gyrus, Heschl's gyrus and associated structures involved with auditory processing and speech perception; the occipital and inferotemporal cortices and the parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition; the precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with the planning, initiation, and execution of motor movements; the dominant hemisphere frontal operculum, dorsalateral frontal cortex, planum temporale and supramarginal gyrus and associated structures involved with language functions; and the networks of structures encompassing the prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention, in learning, and in working memory. Data concerning the user's mental state is then used by the computer system to optimally adapt the information presented by the computer to match the current mental capacity of the user.

The main advantage of the "Sympathetic Neurocognitive Adaptive Computer Interface" is that it automatically adapts the information presented to the user by taking into consideration the level and type of mental effort the user is expending. This differs from other user interfaces where the system responds only to the user's specific, consciously directed commands.

In order to use the system, it must first be calibrated to the individual user. This is accomplished when the system is first operated by a particular user. After that, an abbreviated calibration is performed at the start of each session or as needed. During the calibration, the user performs a brief battery of standard tasks each with several levels of difficulty while her or his behavioral and physiological data are measured. The tasks, such as those described in patent application Ser. No. 07/766,826, Noninvasive Human Neurocognitive Performance Testing Method and System, now U.S. Pat. No. 5,295,491, are designed to cause the user to make a graded series of efforts to engage basic neurocognitive functions such as working memory, divided attention, spatial and grammatical reasoning, etc. Sensory and motor control tasks are also performed. Then the physiological data are analyzed to form indices characteristic of each difficulty level. The EEG data are further analyzed to form topographic templates to determine the pattern of activation of major regional brain systems for each task. These analyses are used to derive a user-specific mathematical function for relating patterns of neuroelectric activity, with or without other physiological measures, to overall mental workload level, and to the degree of utilization of specific brain systems involved with a given task or behavior. The system then operates as follows: As the user is performing a task at the computer, her or his physiological signals are measured and submitted to the user's personal mathematical mental workload function to determine a score related to the user's overall mental workload level, as well as scores for the relative activation of each of the major regional brain systems. In the case of a user controlling a complex system such as an airplane, the system can take over some of the functions being performed by the user if the overall workload score surpasses a high level theshold. Conversely, if a score is below a low level threshold, the system can assign more functions to the user to perform. The system can also adjust the form of information presented to the user based on the relative activation of the major regional brain systems. For example, if the user's spatial processing areas are too occupied, but the overall mental workload is not too high, the system can change the presentation format of some of the tasks from spatial to numeric and linguistic. In the case of a user performing an attention training exercise, the system can present information at the instant at which the user's preparatory attention is optimal as determined by measuring the user's mental effort and regional brain utilization prior to presentation of information. It can then present the information only when level of mental effort and regional brain utilization are at appropriate values, or it can present information in the sense modality or form for which the user's brain is adequately prepared.

Discussion

The results from this study demonstrate the feasibility of using neural network pattern recognition analysis of subject-specific physiological features to distinguish between levels of mental workload unconfounded by such factors as differences between conditions in the amount of eye or motor activity. These results are thus consistent with prior results from our laboratory in which a neural network pattern recognition analysis was used to detect electrophysiological changes related to variations in mental fatigue or vigilance levels (Gevins et al, 1990), in performance accuracy, and in type of cognitive task being performed.

The most important result to emerge from this work is that multiple spectral features of ongoing EEG and EMG can be used to differentiate physiological patterns associated with high and low levels of mental workload with a temporal resolution of less than thirty seconds. Further, the analysis was performed on overlapped sets of trials, which is akin to using a sliding window of data to obtain a continuous estimate of workload, where, within the window, portions of the signal may not be used due to response contamination. This has the important benefit of allowing the assessment of mental workload even during periods in which some of the signals cannot be used due to artifacts; a situation likely to occur when recordings are made in demanding operational environments. These results also highlight the utility of including multiple physiological measures to more sensitively detect workload variations, as well as the utility of tailoring pattern detectors to the idiosyncracies of individual subjects. These issue are considered at more length below.

Improved Classification with Multiple Features and Classification-Directed Feature Selection. The neural network analysis was applied to physiological features alone or in combination and resulted in very simple networks. In all but one case, multiple signal features were required to distinguish the two mental workload conditions. This is not surprising since the task used requires multiple mental resources, such as stimulus recognition, working memory, and motor preparation and control, and the mapping between these and electrophysiological measurements is likely to be quite complex. By contrast, it was surprising that simple networks having only one input unit were sufficient. This result suggests that, with appropriate candidate signal features, the process of selecting the best combination of features based on classifier performance was the most important aspect of the algorithm that was used rather than the power of neural network structures for handling classification problems (Gevins and Morgan, 1986).

The finding that heartrate is not a sensitive index of mental workload in the current situation may be interpreted as indicating that whereas heartrate may be a useful index for physical workload and emotional stress, it is relatively insensitive to variations in cognitive load per se. In contrast, clean EEG features (i.e. those from which possible contamination due to low frequency motor potentials or high frequency muscle activity were eliminated) showed a high classification accuracy, with a time resolution of 80 seconds. Whereas the exact spatial topographies of EEG features which discriminated between workload levels differed across subjects, these features most often involved changes in alpha and/or theta activity (one of the four subjects presented a more complex picture involving all of the analyzed EEG frequency bands to some extent). These results are consistent with other studies demonstrating spatially distributed changes in alpha and theta band activity as tasks vary in cognitive load. In addition to changes in these spectral bands, several recent studies have also shown that sustained slow potential shifts are reliably related to task difficulty or information load. Given that the current study focused on electrophysiological features which could be reliably recorded in operational environments, and that movement artifacts could potentially be mistaken for slow potential shifts, low frequency features were not included in the formal pattern recognition analyses. Even so, it is worthwhile to note that post hoc examination revealed substantial task-related changes in delta band activity for some of the subjects in this study.

Finally, combining EEG and EMG features again resulted in high classification rates but with a much better time resolution, 20 seconds. Although the relative weighting of EMG features differed across subjects, it was an important factor for all. Whereas high classification rates with good time resolution were obtained here by combining only EMG and EEG measures, the addition of other variables such as EKG, respiration and eye activity may prove valuable.

Individual Differences in Workload Sensitive Electrophysiological Features. The results from this study clearly indicate the need to "calibrate" EEG based workload indices based to the idiosyncracies of individual subjects. Although some general effects were found across subjects, such as a decrease in alpha and increase in EMG activity with higher workload, the fine details of the index structure were highly specific to each subject. Such between-subject variability is not surprising given that all cognitive tasks draw upon several neural subsystems, from those subserving sensory perception, through those underlying stimulus evaluation and decision, to those serving response execution. All these systems are influenced by the subject's abilities and prior experience. Subjects are also likely to use different mental strategies to perform the tasks, and these strategies may change over time with task repetition in a different manner for different subjects. Clearly there are many sources from which the individual differences in the electrophysiological patterns could arise. Yet despite the wide variation across subjects, sophisticated analysis techniques which are calibrated upon each individual's data, can find common factors of mental workload even though the particular pattern of expression may differ considerably across subjects.

It seems likely that a maximally sensitive workload index based on electrophysiological measures might also need to be optimized for different types of tasks. The high classification rates we achieved in both the EEG analysis and the combined EEG and EMG analysis support the view that mental workload can be indexed in several ways. The best method may depend on many factors including the spectral regions in which clean signals are available, the cognitive and motor demands of the task, and the generalizability and sensitivity of the index within these contexts. The results reported here illustrate one of the trade-offs that must be considered when designing a generalized mental workload index. EEG data from which the effects of muscle and motor-related activity were removed were analyzed, since it was assumed that such an index, which is sensitive to a broad spectrum of higher cognitive functions, would be more likely to generalize across tasks than would one which is influenced by idiosyncratic perceptual and motor demands of the tasks. Although EEG features were found which distinguished the two workload levels, the time resolution was fourfold smaller than that achieved when EMG features were included. The best practical solution may be a combination of indices, weighted differentially according to situation-specific demands.

Conclusions. This study illustrates the feasibility of employing neural network-based pattern recognition techniques to combinations of physiological features in order to derive sensitive and reliable inferences about the mental workload of individual subjects. Physiological data can be continuously and unobtrusively recorded while users perform their duties. The combination of the specific features used to assess mental workload can be adjusted on an individual basis to determine the most sensitive index for each person. The combination of several measures and the use of overlapped trials in data analysis reduces the detrimental effect of artifacts, and most importantly, changes in workload levels can be discriminated with good temporal resolution.