1994 VR Conference Proceedings

Go to previous article 
Go to next article 
Return to the 1994 VR Table of Contents 


A Preliminary Study Of The Human Interface Using Event-Related Potentials For The Control Of The Virtual Reality System

By: Naohiro Mitsutake, Kiyotaka Hoshiai, Yukiko Sugioka, Yasuhito Yamamoto, Kiyoyuki Yamazaki, Katsuro Okamoto, Takami Yamaguchi
Department of Bio-Medical Engineering,
School of High-Technology for Human Welfare
Tokai University

Hisako Igarashi
Department of Life Science
Himeji Institute of Technology

Jun Noritake
Japan Advanced Institute of Science and Technology
Hokuriku

Atsuya Yoshida
Department of Electronics and Information Science
Faculty of Engineering and Design
Kyoto Institute of Technology

Presented By: N. Mitsutake
Dept. Bio-Medical Engineering
School of High-Technology for Human Welfare, Tokai University
317 Nishino, Numazu, Shizuoka 410-03 Japan
Telephone +81-559-68-1211 (ext 4415)
Fax +81-559-68-1156
E-mail man@fh.u-tokai.ac.jp

ABSTRACT

We recently proposed a "Hyper Hospital" concept constructed in a computer-based electronic network using the virtual reality as a human interface to serve as a potential alternative to the conventional medical care system. We are developing a novel human interface that allows non-verbal communication between the Hyper Hospital and users, particularly severely disabled patients. Our human interface never requires any physical reactions for detecting the user's alternative decisions. This interface adapted the P300 component of Event-related Potentials(ERPs) extracted from electroencephalograms;(EEGs). In order to estimate the feasibility of the P300 components of ERPs, ERPs were recorded from healthy subjects during alphabet discrimination tasks in which subjects were ordered to count the prescribed target stimuli (the uppercase letters "A"or"B"). The P300 components of averaged ERP waveforms were successfully obtained from the target trials. This paper describes how this human interface can detect the selective attention of the subjects without any physical movements and how it can be used as a human interface for computer systems.


Introduction

Recently virtual reality systems have been widely applied to various fields. Although reported applications of the virtualreality technology are mostly limited to recreational and industrial usage, medical applications are thought to be useful particularly for educational purposes. It is needless to say that the horizon extended by the introduction of virtual reality technology is vastly wide and not limited to the fields in which it is currently being applied. Moreover, the applications so far reported are mainly for healthy people and the possibility applying virtual reality technology to help disabled persons has not been considered seriously. Virtual reality technology can be used by disabled people for obtaining various useful information effectively through a virtual experience. It will be possible to compensate for the disordered function of mentally and/or physically handicapped people utilizing this kind of advanced computer technology.

This is why we proposed a new concept of a medical care system named the "Hyper Hospital" which is constructed in a computer based electronic information network[1],[2],[3]. It will be built as a distributed system on the network and consists of all kinds of conventional medical care facilities integrated in the real and the imaginary space. Virtual space of the hyper hospital and private information will be owned and exclusively controlled by the patient him/herself, thus allowing the maximum protection of the patients' privacy. In the hyper hospital, an innovative human interface is required for the consultation, treatment and other communication with the medical staff. This is especially true if we think of severely disabled patients (for example, with the locked-in syndrome[4]) as the user. The purpose of the present study is to establish a new human interface based on the Event Related Potentials (ERPs) which will enable disabled patients to communicate without any physical reactions with other people including medical care staff.

Among many ERP signals, the P300 (positive 300 milliseconds) component of the ERP is well known to be related with higher order brain activity. Farwell and Donchin[5] reported an attempt to use the P300 signal for the character input to a computer system. There are some important points to be solved before being able to actually apply the ERPs or P300 as a control signal to the external system. It is well known that the P300 component of the ERP is observed during performing the oddball tasks in which the task is to detect the infrequent target stimuli from a series of target-nontarget mixed stimuli[6]. A low subjective probability of appearance of the target stimulus is necessary to obtain the P300. Sophisticated signal analysis techniques are also necessary to detect the P300 signal from the EEG measurements. The status of the subjects mind can also be important. For example, a task to count the target stimuli or to press a switch is known to be important to obtain the P300. In the present study, we conducted a human experiment for determining the necessary conditions for utilizing characteristics of the P300 mentioned above for controlling the virtual reality system for disabled people.


METHOD

Six healthy male subjects (mean age = 23yrs, SD = 1.0) participated in the experiment. Subjects were seated on a chair in a sound-attenuated and electrically shielded room. The personal computer (PC9801:NEC) used for stimulus presentation was set in front of the subject at a distance of 45 cm. The procedure of the alphabet discrimination task is as follows. For the task stimulus, four kinds of letters "A","B","a" and "b" (with a height of 60mm and a width of 45mm) were used. One of these letters was selected in a random order and was displayed on a CRT screen for a duration of 300ms. The probability of appearance of the uppercase letters "A" and "B" was each set to 0.10, and that of the lowercase letters "a" and "b" was each set to 0.40 each. Subjects were ordered to count the self prescribed target stimuli (the uppercase letters "A" or "B") in their minds without any physical movements. Four hundred stimuli (letters) were displayed in a session. Although the number of occurrences of the target stimuli was not fixed due to its random manner of appearance, 35 to 45 target characters were shown on the average.

Disk shaped Ag-AgCl electrodes were applied to the scalp at Fz,Cz and Pz regions based on the international 10-20 standard electrodes setting. Linked ear lobes served as the reference, and the electrode impedance was less than 5KOhms. EEG signals were amplified (time constant=1.5s , high frequency cut filter=100Hz) and written out on paper and recorded on a floppy disk with a sampling frequency of 200Hz by a digital data recorder (DRF1:TEAC).

The EEG signal of the length of 500ms was segmented starting from each stimulus (the letter display) onset. The obtained EEG signals were averaged for 160 or 40 times in the case of non-target stimuli or target stimuli, respectively. The segments which were contaminated with serious artifacts due to electrooculograms, and so on, were excluded from the subsequent analysis.


RESULTS

Examples of averaged ERP waveforms in a subject (a 23 year old, male university student) are shown in Fig. 1. The P300 related positive reflection of the averaged EEG wave is seen about 355 ms after the stimulus onset (shown by the arrow in Panel A). The P300 components were identified as the most positive peak in the period of 300 to 400ms after the stimulus onset. The relative zero potential level of the ERP was defined with a mean amplitude of EEG with a pretrigger period of 50ms. The amplitudes of the P300 components were measured as the relative potential from the above mentioned zero point to the peak.

The P300 components are clearly shown in target letter counting trials. In the non-target trials (Panel B, C and D), no significant positive peaks were observed in the period of 300 to 400 ms after the stimulus onset. This tendency of P300 amplitudes was observed in all subjects.

Table 2 indicates peak amplitudes of the P300 components in each experimental condition in all subjects (N=6). Mean amplitudes of the P300 in target trials were significantly larger than those in non-target trials (p<0.05: paired T-test).


DISCUSSION

Event-related Potentials(ERPs) have been used to evaluate the psychological processes during the performance of various tasks[7]. ERPs are mainly divided into two categories. One is exogenous potentials and the other is endogenous potentials. Exogenous potentials reflect obligatory responses evoked by physical features of the stimulus. On the other hand, endogenous potentials reflect cognitive functioning[8].

For this study, we focused on the endogenous potentials of the P300 component. It is well known that the amplitude of the P300 component is associated with the subjective probability of appearance of relevant stimuli, task difficulty, selective attention and so on. In particular, these characteristics can be used to detect the binary decision of the subjects without any physical reactions. The P300 component has potentials as an effective communication tool for establishing a novel human interface for the locked-in syndrome patients.

The present experiment demonstrated that the P300 components can be a potential tool to detect alternative or binary decisions in the mind such as A or B, Yes or No, and so on. In this study, a considerable amount of time was required to detect the subjects decision, at least about 200s. Farwll and Donchin [5] reported their human-computer communication system using the ERPs. They used a 6-by-6 matrix of letters, and rows / columns of the matrix were flashed alternately. Their technique is relatively complicated and is not suitable for use in a virtual reality system which has various kinds of visual objects.

We used a simple method to detect alternative decisions. By repeating this unit decision N times, it is possible to select one from 2n items. Fig. 2 shows a sample menu format with a decision tree type depth indication designed for the use in our virtual reality system. By using this kind of decision tree, a significant amount of time will be necessary to decide one item, if no other additional support system is devised. If we can, however, incorporate decision supporting mechanisms such a semantics analysis system or a front end processor for word recognition, the decision will be rapidly recognized by a computer. Employment of artificial intelligence may be useful to abbreviate the decision tree.

Some attention should be paid to the stability of the ERP measurement and to the mode of the stimulus presentation. A counter measure to the artifacts is also very important. The electrooculogram related activities often contaminated the EEG signals measured particularly from Fz and Cz regions. The EEG signals from Pz were frequently adulterated with alpha waves. Digital signal processing techniques to improve S/N ratio or an adaptive filtering must be considered although there is a tradeoff between the computational time required and the time limitation due to the necessity for real time decision making.

For the practical application of the present method to disabled patients, a high efficiency of communication will be mandatory. In order to improve efficiency, some specialized training of the individual subject may be useful. This training method can include an employment of the biofeedback technique for enhancement of the ERP. This can be useful because we noticed the amplitude of the P300 waveform differed from one subject to another. It may be possible to train a person to show the P300 waveform, and this may be an alternative way toward the practical use of the present method.

In conclusion, the present method to detect the subjective decision without any physical movements by using the P300 components of the ERP was shown to be a potential means to build a novel human interface to computers, particularly by the aid of a virtual reality system.


ACKNOWLEDGMENTS

The authors thank to the following grants and foundations for their support.

  1. Grant from the Telecommunications Advancement Foundation (TAF) for 1993 and 1994, titled "Development of Home Care Support System Using an On-line Virtual Environment."
  2. Grant from the Foundation for Health Facilities Development for 1993, titled "Hyper Hospital, a novel paradigm of the medical care facility expanded on the computer information network interfaced by virtual reality technology."
  3. Grant from the Japan Foundation for Aging and Health, for 1993-1995, titled "A functional training system for aged people using virtual reality system."

We would also like to thank Dr. Tad W. Taylor and Mr. Greg. P. Jewell for their helpful advice and proofreading. We appreciate Miss Nao Yonehana for her help in the preparation of the manuscript.


REFERENCES

  1. A. Yoshida et al., On the Concept of Hyper Hospital, a Medical Care System Distributedly Constructed on the Electronic Information Network, Proceeding of 2nd IEEE International Workshop on Robot and Human Communication, IEEE Catalog Number 93TH0577-7, 365-369(1993).
  2. Y. Hagita et al., Which do you feel comfortable, interview by a real doctor or by a virtual doctor? -A comparative study of responses to inquiries with various psychological intensities, for the development of the Hyper Hospital, Proceeding of 2nd IEEE International Workshop on Robot and Human Communication, IEEE Catalog Number 93TH0577-7, 370-374(1993).
  3. N. Mitsutake et al., Open Sesame from Top of Your Head - An Event Related Potential Based Interface for the Control of the Virtual Reality System, Proceeding of 2nd IEEE International Workshop on Robot and Human Communication, IEEE Catalog Number 93TH0577-7, 292-295(1993).
  4. N. Mauss-Clum et al., Locked-in syndrome : a team approach, The Journal of Neuroscience Nursing, Vol. 23, No. 5, pp 273-85(1991).
  5. L. A. Farwell and E. Donchin., Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials, Electroencephalography and Clinical Neurophysiology Vol. 70, pp. 510-523(1988).
  6. R.F Hink, S.A Hillyard and P.J Benson., Event-related brain potentials and selective attention to acoustic and phonetic cues, Biological Psychology Vol. 6, pp. 1-16(1978).
  7. E. Donchin et al., Application of Brain Event-Related Potentials to Problems in Engineering. Biological Psychology, pp. 702-718(1982).
  8. A.W.K. Gaillard, Problems and Paradigms in ERP Research, Biological Psychology Vol. 26, pp. 91-109(1988).

Go to previous article 
Go to next article 
Return to the 1994 VR Table of Contents 
Return to the Table of Proceedings 


Reprinted with author(s) permission. Author(s) retain copyright.