2001 Conference Proceedings

Go to previous article 
Go to next article 
Return to 2001 Table of Contents

Hands-Free Computer Access For The Severely Disabled

Dr. Andrew M. Junker, Brain Actuated Technologies Inc.
E-mail: admin@brainfingers.com 

Dr. Jane R. Wegner, University of Kansas
E-mail: Janew@dole.isi.ukans,edu 

Dr. Thomas Sudkamp, Wright State University
E-mail: tsudkamp@cs.wright.edu


Results of a study to test the innovative use of the Cyberlink Control System (CCS) as a hands-free input device to access alternative and augmentative communication (AAC) software are presented. This study was conducted under the National Institutes of Health, National Institute of Child Health and Human Development Grant Number 1 R43 HD39070-1. An intervention study was conducted with twenty-five individuals of varying ages and disabilities who had been unable to access AAC technology due to physical limitations. Quantitative data, time history and performance measures, and qualitative data, questionnaires, videotapes and observation, were collected. Subjects were presented with various CCS tasks to evaluate the potential for AAC. Qualitative data was used to assess subjective levels of motor control, cognitive abilities, communication skills, and levels of medication for comparison with the findings from the quantitative data. The data gathered in the study was used to examine the feasibility of enhancing the CCS with the incorporation of learning techniques and adaptive control. Results of this study will be presented.


The field of AAC, though relatively young, has provided a means for many individuals to express themselves that was not available in the past. Alternative computer input devices are now available in a variety of options to support special needs communication and access to AAC (i.e., frontalis muscle switch, head and eye-tracking devices, chin switch, "sip and puff," voice activation, etc.). Use of this technology is, however, dependent on the user’s ability to control the device.

There are individuals, who because of the severity of their physical limitations have been unable to access AAC technology through either direct selection or scanning via a switch. These individuals often have disabilities related to cerebral palsy (CP), amyotrophic lateral sclerosis (ALS), multiple sclerosis (MS), muscular dystrophy (MD), or traumatic brain injury (TBI).

Other practical barriers involve physical fatigue associated with use, length and intensity of required system calibration and adjustment, length and intensity of required user training, and expense. Because of the access difficulties, communication for these individuals is often limited, left to interpretations by communication partners, or non-existent. Given the growth and advances in AAC technology that are taking place, a viable alternative access is the only factor standing between individuals with the most severe disabilities and the technology available to provide independent communication. The CCS has the potential to bridge the gap between these individuals and the communication technology available to empower and transform their lives.

The CCS Solution

The CCS uses brain and body forehead bio-potentials in a novel way to generate multiple signals for computer control inputs. Three silver silver-chloride plated carbon filled plastic sensors in the CCS headband in conjunction with the CCS amplifier circuit can detect bio-potentials as low as 0.3 micro-volts. The forehead site is an ideal measurement site, rich in bio-potentials from a wide range of brain and motor neuronal sources. Signals are easily and non-invasively obtained and impedance levels are usually below 50 K ohms with little or no skin preparation. The forehead is often the last site to suffer degradation in cases of severe disability and degenerative disease. For example, in ALS and MD the ocular motor neurons and ocular muscles are usually spared permitting at least gross eye movements, but not precise eye pointing control.

The CCS derives control signals in three parallel channels from the forehead potentials. The lowest frequency channel, which is defined here as the EOG (Electro-OcularGraphic) signal is responsive to bio-potentials resulting from eye motion. It is band pass derived within the frequency range of 0.2-3.0 Hz. The CCS software (resident in the PC) can accurately differentiate left and rightward eye glances by detecting a phase reversal in this signal. This EOG signal has been used effectively by able-bodied test subjects for left/right computer cursor positioning and discrete switch control (i.e., left mouse button click). Individuals with severe physical limitations such a near-total loss of facial muscle functionality may retain some degree of control over eye motion, which the CCS can exploit.

The second channel is band pass derived between one and 45 Hz, falling within the accepted EEG (ElectroEncephaloGraphic) range. It is sensitive to both prefrontal mental activity and subtle facial muscle inputs and will be referred to here as the ‘Brain-Body’ signal. A patented decoding algorithm subdivides this region into ten component frequency bands in a more timely fashion than was possible previously with the more computationally expensive Fourier Transform. The ten frequency bands are distributed across the brain-body frequency range. The following frequency centers are used: three low frequency bands (centered at 0.95, 2.75 and 4.40 Hz), three mid frequency bands (centered at 7.75, 9.50 and 11.45 Hz) and four high frequency bands (centered at 13.25, 16.50, 21.20 and 25.00 Hz). Bandpass magnitude and frequency shift values are derived from the decoded brain-body signal about each bandpass center frequency. The low frequency bands are sensitive to eye movements, and the high frequency bands are sensitive to forehead muscle activity.

Preliminary studies have shown that continuous control of these frequency bands is often learned first through subtle tensing and relaxing of various muscles including forehead, eye and jaw muscles. After just a few sessions with the CCS, however, many subjects begin to experiment with more efficient, internal EEG-based control methods. This result was of particular interest for the population evaluated in this study, whose physical limitations often preclude the use of facial and precise eye muscle-based access pointing devices.

When the time-averaged magnitude of one of the low frequency bands is mapped to horizontal cursor motion while a high frequency band is mapped to vertical cursor motion, the user can achieve full two-axis cursor control. In preliminary studies one subject, diagnosed as comatose, used the CCS to move through a 50-wall maze. His repeated success influenced his doctor to upgrade his diagnosis and hence his level of care.

Envelope-detected between 70 and 3000 Hz, the third channel is defined as an EMG (ElectroMyoGraphic) signal. It rapidly and accurately responds to subtle contractions of the masseter and frontalis muscles and is well suited to discrete on/off switch closures, keyboard commands, and the functions of the left and right mouse buttons for users with any significant residual of facial muscle functionality. In a study of CCS EMG discrete control conducted at the USAF Armstrong Laboratory, Wright Patterson AFB, Dayton, Ohio with able-bodied subjects, response accuracy was found to be extremely high, approximately 98%. Reaction times fell between 180-200 msec, a range considered to be the limit of simple reaction time. Several subjects actually achieved 15-20% faster reaction times with the CCS EMG discrete command than with a manual button.

The EMG and EOG waveforms of specific facial and eye movement gestures, such as an eyebrow lift, jaw close, rightward or leftward glance, etc. can be discriminated by the CCS software and mapped to separate mouse, keyboard, and program functions.

A filtered version of the EMG is also available as an analog control. This EMG signal can provide an accurate and reliable means of continuous cursor positioning. Users often employ more subtle techniques (for example; light pressure of the tongue against the roof of the mouth) for continuous EMG control while reserving somewhat more demonstrative actions, such as short eyebrow lifts, for the EMG discrete on/off command. For individuals with severely limited control of their facial muscles, the CCS software can be formatted to use Brain-Body or EOG inputs (instead of EMG) to activate switch closures and mouse button clicks.

A CCS Windows 95/98-mouse driver fulfills all the cursor positioning and left/right button functions of the manual mouse and supports ASCII character and string entry. This allows hands-free control of third party WINDOWS applications including augmentative communication and control software such as Words Plus EZ Keys, WiViK2, Clicker 4s, Dynavox For WINDOWS, Gus, Maxhome, and X-Ten.

Special training software helps the user to learn to control the CCS mouse driver through the biofeedback paradigm provided in the venue of on screen visual presentations of the user's brain and body signals and video games such as Pong and Tetris. One-axis continuous tracking studies have shown that CCS users can achieve 90%+ accuracy after a few hours of training. Training for the higher level skills needed for control of Internet browsers, e-mail, CAD programs and other third party WINDOWS 95/98 applications is accomplished through a suite of CCS maze, target acquisition, and music games and utilities progressing from single-axis continuous to multi-axis continuous plus discrete control. Users are provided with training progress performance measures (such as time through a maze, number of balls hit, target acquisition time, etc.) and the ability to adjust and format the interface to fit their individual needs.

Five able-bodied subjects participated in a target acquisition study to evaluate the CCS mouse driver as a hands-free mouse replacement. After four 30-minute training sessions, all five subjects were able to use the CCS mouse driver to position and click the cursor on randomly appearing (Windows icon-sized) targets. Their target acquisition times were often under four seconds and compared favorably with their manual mouse performance on the same task, despite far greater experience with the manual mouse. Several subjects reported that, although it required more conscious effort than the manual mouse initially, control with the CCS mouse driver eventually began to feel more natural and automatic.

Experimental Design and Methods

There were two primary goals pursued in this research effort. One was to determine if the CCS could provide access for individuals who have been unable to obtain access through direct selection or switch controlled scanning or any other alternative control device currently available. The second was to examine the feasibility of enhancing the CCS with the incorporation of learning techniques and adaptive control.

The subjects of this study were twenty-five individuals who had been unable to access technology due to physical limitations. The subjects were recruited from the Lawrence, Topeka, and Kansas City areas through local service providers. Female and male children and adults who have cerebral palsy, ALS, and traumatic brain injury were recruited.

Each of the 25 subjects participated in 3 intervention sessions. Each session lasted approximately one hour. The sessions took place at the University of Kansas Schiefelbusch Clinic or in the subjects' homes. Intervention sessions were videotaped. Time history data and performance scores during subject CCS control were automatically stored electronically for all intervention sessions.

Subjects were exposed to three real-time displays of their forehead bio-potential signals. Each display shows a continuous real time trace of the subject’s EOG, Brain-Body or EMG signals. The brain-body display includes an additional display of the ten component frequency bands in the form of time varying light bars, each bar height is proportional to the magnitude of the related frequency band. Light bars are color coded, blue for low frequency (.95-4.4 Hz bands), green for mid frequency (7.75-11.45 Hz bands), and red for the high frequency range (13.25-25 Hz bands). The subjects were given simple goals such as "try to raise the light bars on the right," or "try to make the EMG signal as small as possible," or "try to move your eyes to the left and back," for example. Subjects were given the opportunity to attempt to bring each of the signals under conscious control. Target trajectories were also presented to the subjects as part of the EMG and EOG displays. The subjects were asked to try to follow the targets.

Subjects were given a further opportunity to attempt to control the 10 frequency bands and the EMG in a simple positional task. The magnitudes of the 10 bands and EMG were displayed on the screen as 11 vertically moving boxes. A target appeared at a randomly selected vertical height at the right of the screen and then slowly moved across the screen. The task was to place the boxes in the path of the moving target.

Results of the above activities provided the interventionist with preliminary information to format the CCS software to the subject for continuous cursor control and discrete switch control. Previous work with the CCS and able-bodied subjects indicated that the 2.75 Hz frequency band would be the best candidate for horizontal cursor control for all subjects. Depending upon the degree of subject motor impairment, either EMG, the 7.75 Hz band, or the 25 Hz frequency band was considered for vertical cursor control.

Subjects were trained in basic one-axis horizontal cursor control using the CCS pong game software. In initial training sessions, subjects were permitted to discover their own abilities and limitations and in effect "fail" in the pong task (i.e., miss the target ball or be unable to move the cursor at all). Once the interventionists observed initial performance, they began adjusting and possibly reformatting the user interface to the subject’s special needs. Subjects were trained similarly in the vertical one-axis CCS pong game. Subjects used a two-axis cursor; formatted according to their best performance formats for horizontal and vertical control as determined above, to navigate through a 50-walled maze. As in other steps the interventionist worked with the subject to adjust and format the CCS interface, where necessary, to aid the subject with system training.

Subjects attempted to control the discrete CCS signals by practicing excitation and relaxation to cross discrete command thresholds or triggers with the CCS "click" and other gesture controlled training utilities. Depending upon user success with the above tasks, subjects were presented with the possibility to control an on-screen keyboard such as Clicker 4 or EzKeys, and an X10 environmental control program in full pointing and click mode and/or in scan and click mode.

Time history data of EOG, Brain-Body, EMG and brain-body frequency band magnitude and frequency shift values were collected at selected times by the interventionist. The subject was instructed to perform a certain function with the CCS such as "keep the EMG signal as low as possible", or "perform a click command when I say go," for example. As the subject began to perform the requested function the interventionist triggered the software to collect and store the time history data. Performance scores resulting from using the various training software programs were recorded as well.

Subjects and families were interviewed and completed a survey concerning the learning, use, and impact of the CCS. The survey focused on obtaining measures relative to the subjects’ degree of motor control and communication skills prior to working with the CCS. Information of types of medications and affects on motor control, and options on the strengths and weaknesses of the CCS, and possible impact of CCS access on the individual's quality of life were obtained.

The qualitative data obtained from interviews, surveys, and journals was reviewed, organized and tabulated for comparisons across subjects. The video data was analyzed across and within subjects. Descriptive statistics was used to describe the data. Taken collectively across subjects, the data reflected experiences common to the group

The training performance data was analyzed across and within subjects. Interface adjustment and formatting values were tabulated across and within subjects over the three sessions. Trends and relationships between disability, training performance and adjustment/format settings were evaluated.

The data gathered in the study was used to examine the feasibility of enhancing the CCS with the incorporation of learning techniques and adaptive control taken from Fuzzy Set Theory. Fuzzy Set Theory provides a foundation for controlling complex systems in that the information being processed is imprecise or inaccurate. The ability to have a robust control system for the CCS is extremely desirable in applications for persons with disabilities. In this study, subjects found it difficult to produce and/or repeat all of the facial and eye movements necessary to control the CCS. Consequently, the input received by the system consisted of imprecise patterns of movement and signals with noise.

Historically, fuzzy modeling and control systems were constructed from human expertise and heuristic knowledge of the system being modeled. As the systems being modeled have increased in complexity, learning algorithms have been developed to produce models from training data. A particular problem for constructing a fuzzy controller for the CCS was the sparseness of data from which the controller could be built; most combinations of possible inputs did not always occur and could not be recorded in the user training sessions. Interpolation techniques, known as completion algorithms, have been developed to extend the scope of partial training information. Constructing a controller based upon the performance exhibited by individual introductory training sessions was undertaken that would have the ability to reduce the length of initial training periods.

Ideally, a CCS controller should be able to adapt based upon changing characteristics of the user. Adaptation includes long-term modifications to reflect improved capability gained with experience and short-term modifications to adjust to minor changes in performance. Fuzzy adaptive control strategies that have been developed to compensate for both types of changes in the user's performance were implemented. The incorporation of time-series prediction of fuzzy information into the CCS was also undertaken. Prediction by the recognition of a general pattern of actions (intermixed with noise and possible spurious information produced by inadvertent movements) has the potential to increase reliability and system response time.

Results To Be Presented

Results of this study will be presented. The potential benefits from the incorporation of the learning and adaptive control algorithms will be discussed. Focus will be in terms of the reduction of the length and intensity of user training, the development of a system more tailored to the capabilities of the individual, and the development of a system that will adapt to compensate for changes in the user's performance

Go to previous article 
Go to next article 
Return to 2001 Table of Contents 
Return to Table of Proceedings

Reprinted with author(s) permission. Author(s) retain copyright.