Patent No. 5649061 Device and method for estimating a mental decision

 

Patent No. 5649061

Device and method for estimating a mental decision (Smyth, Jul 15, 1997)

Abstract

A device and method for estimating a mental decision to select a visual cue from the viewer's eye fixation and corresponding single event evoked cerebral potential. The device comprises an eyetracker, an electronic biosignal processor and a digital computer. The eyetracker determines the instantaneous viewing direction from oculometric measurements and a head position and orientation sensor. The electronic processor continually estimates the cerebral electroencephalogramic potential from scalp surface measurements following corrections for electrooculogramic, electromyogramic and electrocardiogramic artifacts. The digital computer analyzes the viewing direction data for a fixation and then extracts the corresponding single event evoked cerebral potential. The fixation properties, such as duration, start and end pupil sizes, end state (saccade or blink) and gaze fixation count, and the parametric representation of the evoked potential are all inputs to an artificial neural network for outputting an estimate of the selection interest in the gaze point of regard. The artificial neural network is trained off-line prior to application to represent the mental decisions of the viewer. The device can be used to control computerized machinery from a video display by ocular gaze point of regard alone, by determining which visual cue the viewer is looking at and then using the estimation of the task-related selection as a selector switch.

Notes:

Device for estimating a mental decision. Filed May 1995, granted July 1997. Assigned to Army. As applied to the remote monitoring, the filtering process mentioned would be how the attackers (human or computer) would/could be able to ascertain the line of sight, visual fixation or even the pupil sizes. This goes back to the action potentials. This patent also goes into extracting single evoked potentials from a field of view, or if you prefer a plurality of evoked potentials. Goes into regression analysis for determination of specific evoked potentials (means using previously monitored and recorded evoked potentials and comparing them to determine what a specific evoked potential translates to). Background of the invention, paragraph 2 omitted use as a 'trigger' type mechanism in a torture routine in a computerized or a human controlled torture device.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to estimating a mental decision to activate a task related function which is selected by a visual cue and, therefore, can be used to control machines from a visual display by eye gaze alone.

The present invention has many potential applications in the medical, scientific, engineering, manufacturing, military, entertainment, and other fields. The present invention may be used as a tool for medical diagnosis of ocular functions, as an aid to the paraplegic handicapped, as an instrument for providing measurement of ocular functions and workload in human factors studies, as a measure of subject training, as a tool for fatigue monitoring, as part of an electronic safety net to detect performance degradation due to pilot incapacitation in piloted and teleoperated vehicles, as a component of an electronic intelligent pilot-vehicle interface used for situation awareness aiding in piloted and teleoperated vehicles, as a tool for task scan analysis including situation awareness measuring, as a controller of machines and computer games, and for advertisement and usability analysis.

Particularly, the present invention is utilized to control computerized machines from an electronic video display by the ocular gaze point of regard alone. Examples of machine control by ocular functions are: (1) updating computer generated information displays, (2) selecting panel switches and instruments, (3) controlling the fidelity of computer generated imagery scene inserts in simulations, (4) controlling the viewing direction of remotely located cameras, (5) controlling the movement of teleoperated robotics platforms or vehicles, (6) selecting display subareas for automated scene analysis in aided target recognition, (7) designating targets from direct sight or from a sensor display, and (8) weapon system pointing.

The present invention has particular applications to time shared concurrent tasks where hands are involved in a continual time critical pilotage task and eyes are used intermittently to control a discrete task. The present invention enables both tasks to share a common visual working area with overlaid visual images. Therefore, the present invention allows task interference to be reduced by dedicating eye-movements and visual attention to the same working surface.

An example of such an application would be single pilot nap-of-earth low-level helicopter flight while updating on-board heads-up displays. A similar application is teleoperations of remote vehicles from video displays with camera control. Another such application is to the operation of completely enclosed armored vehicles with "transparent" or "see through" armor. There the operator would see a video projection of the outside scene, recorded by externally mounted cameras and relayed to internal monitors. The operator would use the present invention to control displays overlaid on the scene projection while concurrently performing the vehicle pilotage task. Similar comments apply to the piloting of "glass cockpit" designs for completely enclosed, high performance aircraft.

The present invention with a properly designed oculometer can also be used with head-mounted video displays in many application fields. The head-mounted video displays, such as those developed for virtual reality, stereographic displays, monocular or binocular vision helmet mounted displays, and night vision goggles used in piloted helicopters, vehicles, and teleoperated robotics control stations are all suitable.

2. Description of the Background Art

The conventional method of decision estimation for control of machines by ocular functions is based on measuring the eye gaze fixation duration, which commonly is longer for a visual cue of task interest than for a visual cue of disinterest. However, the statistical distributions of the interest and disinterest fixation durations tend to overlap and are clearly not separable. In practice, the user must extend the gaze duration with an unnaturally directed stare to designate a cue of interest. Usually, the user follows the display response to the cue selection with a motor action like a button push to confirm the visual selection. In some designs the user must execute an extended stare for cue selection in conjunction with the motor action to indicate a selection. This need for an extensively forced gaze tends to interrupt the task flow since any naturally occurring visual search patterns are momentarily suppressed. It can also increase ocular fatigue due to the corresponding reduction in eye blinks and the associated reductions in corneal lubrication and oxygenation. Furthermore, the need for a confirming motor action increases the workload of the viewer.

SUMMARY OF THE INVENTION

The present invention estimates a mental decision to select a visual cue of task related interest, from both eye fixation and the associated single event evoked cerebral potential. The invention uses the start of the eye fixation to trigger the computation of the corresponding evoked cerebral potential.

The design of the invention is based on the physiological evidence for the relationships existing between eye movements, evoked visual potentials and human visual information processing. This evidence is summarized as follows:

(1) While a visual cue is acquired by a rapid eye-movement known as a saccade, the cue can only be studied during a fixation lasting typically from 200 to 600 milliseconds;

(2) Simple target cues at known locations are identified within several fixations, which taken together define a gaze with the first fixation locating a critical feature of the cue and the second fixation being longer in time during which the critical feature is mentally compared;

(3) The duration of an eye fixation occurring during a mental comparison tends to be longer for a visual cue of task-interest than for a visual cue of disinterest;

(4) The evoked visual potential generated during a fixation period has waveform components which correspond to the stages of visual information processing;

(5) The amplitude of the evoked potential component occurring about 250 to 300 milliseconds is greater for a visual cue of task-interest than for a visual cue of disinterest;

(6) The eye pupil of an alert viewer in constant illumination tends to dilate at the start of information processing reaching its maximum just before decision is made, and contracts at the moment of decision making; and

(7) Eye blinks appear to mark brief breaks that the brain takes at the end of each phase of a mental task, and in this way punctuate the sequence of mental events involved in acquiring and processing information.

A preferred embodiment of the present invention is directed to a device for estimating a mental decision, comprising an eyetracker, an electronic biosignal processor and a digital computer. The eyetracker determines the instantaneous pupil size and line-of-sight from oculometric measurements and a head position and orientation sensor. The biosignal processor continually estimates the cerebral electroencephalogramic potential from scalp surface measurements, following corrections for electrooculogramic, electromyogramic and electrocardiogramic artifacts. The digital computer uses an expert system program to first segregate in real time an eye fixation from the line-of-sight data and then extracts and parameterizes the corresponding single event evoked visual potential from the electroencephalogramic data. The expert system program computes the fixation start, fixation end and pupil sizes. It classifies the fixation end state as a saccade or an eye blink and groups the fixation with prior ones into a gaze point. The eye fixation properties, such as fixation-duration, pupil size change, end state and gaze fixation number, and the parametric representation of the evoked potential, are used as input to an artificial neural network, which outputs an estimate of the mental selection being made.

Conceptually, the present invention functions as a prediction machine separating the two possible visual cue selections, "select" or "reject", in a multivariate dimensional feature set space defined by the eye fixation properties and the parametric representation of the concurrent single event evoked potential generated during the visual information processing. In this way, the present invention makes use of the naturally occurring physiological processes that underline the task related decision making process, since these physiological processes are readily generated as the user looks over the scene to select display control cues. That is, the measurements of both eye fixation properties and the associated single event evoked cortical potential are used to improve the representation of the task related decision making process. This results in a more natural, task related decision making procedure with the reduced number of forced gazes required for task related selections.

Moreover, the present invention can function to control computerized machinery from a video display by ocular gaze point of regard alone. This task is accomplished by determining which visual cue the viewer is looking at and using the estimation of the task-related selection as a selector switch. In addition, it offers auto-calibration since the user can quickly and accurately go through a calibration procedure that correlates selector decisions with ocular and physiological measurements. Then the artificial neural network is trained off-line prior to application to represent the decision process of the viewer.

Further scope of applicability of the present invention will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art from this detailed description.

The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.