2001 Conference Proceedings
Go to previous article
Go to next article
Return to 2001 Table of Contents
Hands-Free Computer Access For The Severely Disabled
Dr. Andrew M. Junker, Brain Actuated Technologies Inc.
E-mail: admin@brainfingers.com
Dr. Jane R. Wegner, University of Kansas
E-mail: Janew@dole.isi.ukans,edu
Dr. Thomas Sudkamp, Wright State University
E-mail: tsudkamp@cs.wright.edu
Introduction
Results of a study to test the innovative use of the Cyberlink
Control System (CCS) as a hands-free input device to access
alternative and augmentative communication (AAC) software are
presented. This study was conducted under the National Institutes
of Health, National Institute of Child Health and Human
Development Grant Number 1 R43 HD39070-1. An intervention study
was conducted with twenty-five individuals of varying ages and
disabilities who had been unable to access AAC technology due to
physical limitations. Quantitative data, time history and
performance measures, and qualitative data, questionnaires,
videotapes and observation, were collected. Subjects were
presented with various CCS tasks to evaluate the potential for
AAC. Qualitative data was used to assess subjective levels of
motor control, cognitive abilities, communication skills, and
levels of medication for comparison with the findings from the
quantitative data. The data gathered in the study was used to
examine the feasibility of enhancing the CCS with the
incorporation of learning techniques and adaptive control.
Results of this study will be presented.
Significance
The field of AAC, though relatively young, has provided a means
for many individuals to express themselves that was not available
in the past. Alternative computer input devices are now available
in a variety of options to support special needs communication
and access to AAC (i.e., frontalis muscle switch, head and
eye-tracking devices, chin switch, "sip and puff," voice
activation, etc.). Use of this technology is, however, dependent
on the user’s ability to control the device.
There are individuals, who because of the severity of their
physical limitations have been unable to access AAC technology
through either direct selection or scanning via a switch. These
individuals often have disabilities related to cerebral palsy
(CP), amyotrophic lateral sclerosis (ALS), multiple sclerosis
(MS), muscular dystrophy (MD), or traumatic brain injury (TBI).
Other practical barriers involve physical fatigue associated
with use, length and intensity of required system calibration and
adjustment, length and intensity of required user training, and
expense. Because of the access difficulties, communication for
these individuals is often limited, left to interpretations by
communication partners, or non-existent. Given the growth and
advances in AAC technology that are taking place, a viable
alternative access is the only factor standing between
individuals with the most severe disabilities and the technology
available to provide independent communication. The CCS has the
potential to bridge the gap between these individuals and the
communication technology available to empower and transform their
lives.
The CCS Solution
The CCS uses brain and body forehead bio-potentials in a novel
way to generate multiple signals for computer control inputs.
Three silver silver-chloride plated carbon filled plastic sensors
in the CCS headband in conjunction with the CCS amplifier circuit
can detect bio-potentials as low as 0.3 micro-volts. The forehead
site is an ideal measurement site, rich in bio-potentials from a
wide range of brain and motor neuronal sources. Signals are
easily and non-invasively obtained and impedance levels are
usually below 50 K ohms with little or no skin preparation. The
forehead is often the last site to suffer degradation in cases of
severe disability and degenerative disease. For example, in ALS
and MD the ocular motor neurons and ocular muscles are usually
spared permitting at least gross eye movements, but not precise
eye pointing control.
The CCS derives control signals in three parallel channels from
the forehead potentials. The lowest frequency channel, which is
defined here as the EOG (Electro-OcularGraphic) signal is
responsive to bio-potentials resulting from eye motion. It is
band pass derived within the frequency range of 0.2-3.0 Hz. The
CCS software (resident in the PC) can accurately differentiate
left and rightward eye glances by detecting a phase reversal in
this signal. This EOG signal has been used effectively by
able-bodied test subjects for left/right computer cursor
positioning and discrete switch control (i.e., left mouse button
click). Individuals with severe physical limitations such a
near-total loss of facial muscle functionality may retain some
degree of control over eye motion, which the CCS can
exploit.
The second channel is band pass derived between one and 45 Hz,
falling within the accepted EEG (ElectroEncephaloGraphic) range.
It is sensitive to both prefrontal mental activity and subtle
facial muscle inputs and will be referred to here as the
‘Brain-Body’ signal. A patented decoding algorithm
subdivides this region into ten component frequency bands in a
more timely fashion than was possible previously with the more
computationally expensive Fourier Transform. The ten frequency
bands are distributed across the brain-body frequency range. The
following frequency centers are used: three low frequency bands
(centered at 0.95, 2.75 and 4.40 Hz), three mid frequency bands
(centered at 7.75, 9.50 and 11.45 Hz) and four high frequency
bands (centered at 13.25, 16.50, 21.20 and 25.00 Hz). Bandpass
magnitude and frequency shift values are derived from the decoded
brain-body signal about each bandpass center frequency. The low
frequency bands are sensitive to eye movements, and the high
frequency bands are sensitive to forehead muscle activity.
Preliminary studies have shown that continuous control of these
frequency bands is often learned first through subtle tensing and
relaxing of various muscles including forehead, eye and jaw
muscles. After just a few sessions with the CCS, however, many
subjects begin to experiment with more efficient, internal
EEG-based control methods. This result was of particular interest
for the population evaluated in this study, whose physical
limitations often preclude the use of facial and precise eye
muscle-based access pointing devices.
When the time-averaged magnitude of one of the low frequency
bands is mapped to horizontal cursor motion while a high
frequency band is mapped to vertical cursor motion, the user can
achieve full two-axis cursor control. In preliminary studies one
subject, diagnosed as comatose, used the CCS to move through a
50-wall maze. His repeated success influenced his doctor to
upgrade his diagnosis and hence his level of care.
Envelope-detected between 70 and 3000 Hz, the third channel is
defined as an EMG (ElectroMyoGraphic) signal. It rapidly and
accurately responds to subtle contractions of the masseter and
frontalis muscles and is well suited to discrete on/off switch
closures, keyboard commands, and the functions of the left and
right mouse buttons for users with any significant residual of
facial muscle functionality. In a study of CCS EMG discrete
control conducted at the USAF Armstrong Laboratory, Wright
Patterson AFB, Dayton, Ohio with able-bodied subjects, response
accuracy was found to be extremely high, approximately 98%.
Reaction times fell between 180-200 msec, a range considered to
be the limit of simple reaction time. Several subjects actually
achieved 15-20% faster reaction times with the CCS EMG discrete
command than with a manual button.
The EMG and EOG waveforms of specific facial and eye movement
gestures, such as an eyebrow lift, jaw close, rightward or
leftward glance, etc. can be discriminated by the CCS software
and mapped to separate mouse, keyboard, and program functions.
A filtered version of the EMG is also available as an analog
control. This EMG signal can provide an accurate and reliable
means of continuous cursor positioning. Users often employ more
subtle techniques (for example; light pressure of the tongue
against the roof of the mouth) for continuous EMG control while
reserving somewhat more demonstrative actions, such as short
eyebrow lifts, for the EMG discrete on/off command. For
individuals with severely limited control of their facial
muscles, the CCS software can be formatted to use Brain-Body or
EOG inputs (instead of EMG) to activate switch closures and mouse
button clicks.
A CCS Windows 95/98-mouse driver fulfills all the cursor
positioning and left/right button functions of the manual mouse
and supports ASCII character and string entry. This allows
hands-free control of third party WINDOWS applications including
augmentative communication and control software such as Words
Plus EZ Keys, WiViK2, Clicker 4s, Dynavox For WINDOWS, Gus,
Maxhome, and X-Ten.
Special training software helps the user to learn to control the
CCS mouse driver through the biofeedback paradigm provided in the
venue of on screen visual presentations of the user's brain and
body signals and video games such as Pong and Tetris. One-axis
continuous tracking studies have shown that CCS users can achieve
90%+ accuracy after a few hours of training. Training for the
higher level skills needed for control of Internet browsers,
e-mail, CAD programs and other third party WINDOWS 95/98
applications is accomplished through a suite of CCS maze, target
acquisition, and music games and utilities progressing from
single-axis continuous to multi-axis continuous plus discrete
control. Users are provided with training progress performance
measures (such as time through a maze, number of balls hit,
target acquisition time, etc.) and the ability to adjust and
format the interface to fit their individual needs.
Five able-bodied subjects participated in a target acquisition
study to evaluate the CCS mouse driver as a hands-free mouse
replacement. After four 30-minute training sessions, all five
subjects were able to use the CCS mouse driver to position and
click the cursor on randomly appearing (Windows icon-sized)
targets. Their target acquisition times were often under four
seconds and compared favorably with their manual mouse
performance on the same task, despite far greater experience with
the manual mouse. Several subjects reported that, although it
required more conscious effort than the manual mouse initially,
control with the CCS mouse driver eventually began to feel more
natural and automatic.
Experimental Design and Methods
There were two primary goals pursued in this research effort. One
was to determine if the CCS could provide access for individuals
who have been unable to obtain access through direct selection or
switch controlled scanning or any other alternative control
device currently available. The second was to examine the
feasibility of enhancing the CCS with the incorporation of
learning techniques and adaptive control.
The subjects of this study were twenty-five individuals who had
been unable to access technology due to physical limitations. The
subjects were recruited from the Lawrence, Topeka, and Kansas
City areas through local service providers. Female and male
children and adults who have cerebral palsy, ALS, and traumatic
brain injury were recruited.
Each of the 25 subjects participated in 3 intervention sessions.
Each session lasted approximately one hour. The sessions took
place at the University of Kansas Schiefelbusch Clinic or in the
subjects' homes. Intervention sessions were videotaped. Time
history data and performance scores during subject CCS control
were automatically stored electronically for all intervention
sessions.
Subjects were exposed to three real-time displays of their
forehead bio-potential signals. Each display shows a continuous
real time trace of the subject’s EOG, Brain-Body or EMG
signals. The brain-body display includes an additional display of
the ten component frequency bands in the form of time varying
light bars, each bar height is proportional to the magnitude of
the related frequency band. Light bars are color coded, blue for
low frequency (.95-4.4 Hz bands), green for mid frequency
(7.75-11.45 Hz bands), and red for the high frequency range
(13.25-25 Hz bands). The subjects were given simple goals such as
"try to raise the light bars on the right," or "try to make the
EMG signal as small as possible," or "try to move your eyes to
the left and back," for example. Subjects were given the
opportunity to attempt to bring each of the signals under
conscious control. Target trajectories were also presented to the
subjects as part of the EMG and EOG displays. The subjects were
asked to try to follow the targets.
Subjects were given a further opportunity to attempt to control
the 10 frequency bands and the EMG in a simple positional task.
The magnitudes of the 10 bands and EMG were displayed on the
screen as 11 vertically moving boxes. A target appeared at a
randomly selected vertical height at the right of the screen and
then slowly moved across the screen. The task was to place the
boxes in the path of the moving target.
Results of the above activities provided the interventionist
with preliminary information to format the CCS software to the
subject for continuous cursor control and discrete switch
control. Previous work with the CCS and able-bodied subjects
indicated that the 2.75 Hz frequency band would be the best
candidate for horizontal cursor control for all subjects.
Depending upon the degree of subject motor impairment, either
EMG, the 7.75 Hz band, or the 25 Hz frequency band was considered
for vertical cursor control.
Subjects were trained in basic one-axis horizontal cursor
control using the CCS pong game software. In initial training
sessions, subjects were permitted to discover their own abilities
and limitations and in effect "fail" in the pong task (i.e., miss
the target ball or be unable to move the cursor at all). Once the
interventionists observed initial performance, they began
adjusting and possibly reformatting the user interface to the
subject’s special needs. Subjects were trained similarly in
the vertical one-axis CCS pong game. Subjects used a two-axis
cursor; formatted according to their best performance formats for
horizontal and vertical control as determined above, to navigate
through a 50-walled maze. As in other steps the interventionist
worked with the subject to adjust and format the CCS interface,
where necessary, to aid the subject with system training.
Subjects attempted to control the discrete CCS signals by
practicing excitation and relaxation to cross discrete command
thresholds or triggers with the CCS "click" and other gesture
controlled training utilities. Depending upon user success with
the above tasks, subjects were presented with the possibility to
control an on-screen keyboard such as Clicker 4 or EzKeys, and an
X10 environmental control program in full pointing and click mode
and/or in scan and click mode.
Time history data of EOG, Brain-Body, EMG and brain-body
frequency band magnitude and frequency shift values were
collected at selected times by the interventionist. The subject
was instructed to perform a certain function with the CCS such as
"keep the EMG signal as low as possible", or "perform a click
command when I say go," for example. As the subject began to
perform the requested function the interventionist triggered the
software to collect and store the time history data. Performance
scores resulting from using the various training software
programs were recorded as well.
Subjects and families were interviewed and completed a survey
concerning the learning, use, and impact of the CCS. The survey
focused on obtaining measures relative to the subjects’
degree of motor control and communication skills prior to working
with the CCS. Information of types of medications and affects on
motor control, and options on the strengths and weaknesses of the
CCS, and possible impact of CCS access on the individual's
quality of life were obtained.
The qualitative data obtained from interviews, surveys, and
journals was reviewed, organized and tabulated for comparisons
across subjects. The video data was analyzed across and within
subjects. Descriptive statistics was used to describe the data.
Taken collectively across subjects, the data reflected
experiences common to the group
The training performance data was analyzed across and within
subjects. Interface adjustment and formatting values were
tabulated across and within subjects over the three sessions.
Trends and relationships between disability, training performance
and adjustment/format settings were evaluated.
The data gathered in the study was used to examine the
feasibility of enhancing the CCS with the incorporation of
learning techniques and adaptive control taken from Fuzzy Set
Theory. Fuzzy Set Theory provides a foundation for controlling
complex systems in that the information being processed is
imprecise or inaccurate. The ability to have a robust control
system for the CCS is extremely desirable in applications for
persons with disabilities. In this study, subjects found it
difficult to produce and/or repeat all of the facial and eye
movements necessary to control the CCS. Consequently, the input
received by the system consisted of imprecise patterns of
movement and signals with noise.
Historically, fuzzy modeling and control systems were
constructed from human expertise and heuristic knowledge of the
system being modeled. As the systems being modeled have increased
in complexity, learning algorithms have been developed to produce
models from training data. A particular problem for constructing
a fuzzy controller for the CCS was the sparseness of data from
which the controller could be built; most combinations of
possible inputs did not always occur and could not be recorded in
the user training sessions. Interpolation techniques, known as
completion algorithms, have been developed to extend the scope of
partial training information. Constructing a controller based
upon the performance exhibited by individual introductory
training sessions was undertaken that would have the ability to
reduce the length of initial training periods.
Ideally, a CCS controller should be able to adapt based upon
changing characteristics of the user. Adaptation includes
long-term modifications to reflect improved capability gained
with experience and short-term modifications to adjust to minor
changes in performance. Fuzzy adaptive control strategies that
have been developed to compensate for both types of changes in
the user's performance were implemented. The incorporation of
time-series prediction of fuzzy information into the CCS was also
undertaken. Prediction by the recognition of a general pattern of
actions (intermixed with noise and possible spurious information
produced by inadvertent movements) has the potential to increase
reliability and system response time.
Results To Be Presented
Results of this study will be presented. The potential benefits
from the incorporation of the learning and adaptive control
algorithms will be discussed. Focus will be in terms of the
reduction of the length and intensity of user training, the
development of a system more tailored to the capabilities of the
individual, and the development of a system that will adapt to
compensate for changes in the user's performance
Go to previous article
Go to next article
Return to 2001 Table of Contents
Return to Table of
Proceedings
Reprinted with author(s) permission. Author(s) retain copyright.