Go to previous article
Go to next article
Return to 1995 VR Table of Contents
Michael R. Clark
Apple Computer, Inc., Advanced Technology Group
131 S. Barrington Avenue #200
Los Angeles, California 90049
Conventional computer display products for the visually impaired limit the amount of enlarged imagery that can be displayed on a desktop monitor to as little as 1% of a document page. Users also suffer disorientation with respect to cursor position on the page, and face unwieldy methods for navigating about the scene. These problems prevent productive computer access for over 1,000,000 Americans, and are highlighted as a major improvement area in the National Eye Institute's 1994 National Plan.
This paper describes a prototype system designed to address these problems by creating a more natural and intuitive visual interface for enlarged computer imagery. This virtual computer monitor (VCM) moves the user's line of sight across an enlarged virtual document, instead of vice-versa, using a virtual reality head-mounted display (HMD). The wearer's head position is sensed using a head-tracking device, and translated into mouse output in software. The simulated mouse data is used to scan the enlarged computer output across the HMD field-of-view in the opposite direction from the wearer's physical head movement, causing the impression of moving one's view about a fixed, enlarged document.
The VCM prototype is currently assembled and functional using commercially available hardware, including a General Reality Company CyberEyeª HMD and a portable Apple Macintosh computer. In addition to discussing the prototype system, this paper also describes a VCM research and evaluation effort now being initiated under a Department of Health and Human Service research grant, and discusses technical/performance issues requiring further investigation.
Among the visually-impaired population, the most common approach to computer access is specialized software and/or hardware that magnifies the image on the computer monitor. This is because simpler solutions such as moving closer to the monitor, using a larger monitor, adding a screen magnifier, or using a spectacle-mounted telescopic system provide either limited magnification or a very limited viewing field.
These computer display magnification solutions operate by magnifying the original image of the computer screen to a "virtual screen" whose size is much larger than the physical monitor. The visually-impaired user then operates the computer by using a mouse, joystick, or cursor keys to control which portion of the virtual screen is shown on the physical monitor at any one time. Of course, since the monitor is fixed, the user is in essence moving the virtual screen across the monitor as if the virtual screen was a real page on a CCTV system's x/y table.
Unfortunately, There are two basic shortcomings to this approach. The first problem is spatial orientation, in that it is difficult to determine where on the page one's view is directed at any given time. This problem is exacerbated for high magnifications. For example, one study found mean magnifications of 15.48x for nearly 100 experienced CCTV users. At 15x magnification, a 15" monitor can only display about 1% of a standard 8.5"x11" page, making most computer work essentially impossible. The second problem is dynamic control, in that all of the various control schemes for navigating about the page are cumbersome, confusing, and slow. Together, these problems were termed the "field navigation" problem in the National Advisory Eye Council's 1994-98 National Plan, in which the Low Vision and its Rehabilitation Panel identified "text navigation" as a particularly promising opportunity for new technologies.
General Reality Company, with support from Apple Computer, Inc., is now attacking the field navigation problem using a combination of technologies developed for operating within virtual reality environments. Specifically, General Reality is working to design, prototype, and evaluate a system that replaces the visually impaired user's physical computer monitor with a "virtual computer monitor" (VCM) of essentially unlimited size.
In a virtual reality system, the user is "immersed" in a synthetic environment, in which virtual objects can be located anywhere in the user's physical space. The user views these objects by wearing a head- mounted display (HMD) such as the General Reality CyberEye. The user also wears a head-tracker, which senses the direction the user is facing, and sends this information to the host computer. The computer uses this data to generate graphics corresponding to the user's line of sight in the virtual environment.
While virtual reality involves exotic graphics rendering hardware, specialized software, and long application integration cycles, the approach of closing a control loop between head-tracker data and a head- mounted video display can be implemented in a rather straightforward manner for viewing arbitrary computer display data at a wide range of magnifications. The result is a VCM.
Conceptually, a VCM operates by fixing the enlarged virtual screen of data in space and scanning the user's line of sight across the data, instead of scanning the virtual screen across the display device as done in conventional screen enlargement systems. In an ideal implementation, the user should perceive an effect similar to one created by sitting in the front row of a movie theater and turning to look at various points on the movie screen. This approach holds promise as a solution to the field navigation problem because it creates an intuitive navigational technique - to view the upper left corner of the computer display, the user scans her head across the virtual image until she reaches the upper left, while to view the lower right, she turns her head to the lower right. By incorporating a cursor at a fixed position in the user's visual field, interaction with computer data is then possible by turning one's head to the desired insertion point and clicking a mouse button.
General Reality Company and Apple Computer have constructed a breadboard prototype of the VCM interface, using experimental software that translates head-tracker output into mouse input usable by standard text enlargement programs.
The prototype hardware consists of a CyberEye HMD from General Reality Company, a proprietary head tracking system, and a Macintosh portable computer. The computer is a Duo 230, unmodified except that the standard monitor has been removed to reduce weight and battery drain.
The CyberEye Model CE-100 HMD used in the VCM is designed for professional virtual reality and data display applications, and provides 420x230 display pixels over a 22.5 by 16.9 degree field-of-view, focused at near-infinity. This results in individual RGB pixels which subtend about 3.2 arc-minutes as seen by the user. The display uses one 0.7" diagonal full-color active matrix liquid crystal display (AMLCD) for each eye, with high-brightness backlighting and precision injection- molded optics.
In order to convert the Macintosh Duo's VGA display output into composite NTSC video format acceptable to the HMD, a portable docking station known as the Powerlink Presentor is used. This docking station also provides an Apple Desktop Bus (ADB) port which is used to input tracking data, and a line-level audio output connector used to provide audio to the HMD's headphones. The ADB port is connected to a small (approximately 6" by 3" by 2") box containing support electronics for the HMD and tracker. All power required by the peripherals is supplied by the ADB port, which simplifies operation and enables mobile use of the prototype without tethering wires.
The head-tracker is a proprietary prototype of a two-degree-of-freedom tracking system which uses a pair of orthogonally mounted electronic gyroscopes to sense changes in orientation in the pitch (look up/down) and yaw (turn left/right) axes. The gyroscopes operate at a lag of approximately 10msec, which supports a design goal of 40msec maximum lag between head movement and updated display video. Recent work at the University of North CarolinaÕs virtual reality laboratory has shown that total system lags of this magnitude are required for augmented reality applications (in which virtual objects are visually overlaid on physical objects using a see-through HMD), and it is postulated here that a similar performance requirement applies to the VCM application.
In operation, InLarge screen magnification software is used to magnify the output of any InLarge-compatible computer application to a user- selected size. For example, with a magnification of 10, a standard 8.5" x 11" page is approximately 7 feet wide by 9 feet tall. Assuming a nominal viewing distance of 24" between a user's eyes and a conventional monitor, The standard CyberEye's 22.5 degree wide field-of-view equates to a 9.5" wide viewing window, which can be located anywhere on the virtual screen by simply looking in that direction. Later in the project a CyberEye HMD will be modified to double its field of view, to create a 19" wide instantaneous viewing window.
It is important to note that instead of developing a maximum-performance research prototype at great expense and later investigating production- related cost reductions, the developers are purposefully using low- priced commercially available components wherever possible. This is being done to ensure that once proven, the VCM can be brought to market quickly at a price that makes computers financially accessible to potential users instead of just technically so. The approximate current retail costs of the major components are computer $2,000, HMD $2,000, and mini-dock $500, with the prototype tracker costing about $1,500. The developers expect that in relatively small production lots of 100 units, the entire system including the computer could retail below $5,000 by the end of 1996.
It is also important to note that the program InLarge is being used at present, although the software interface is designed to be generic. By interfacing to existing screen enlargement software in a standardized manner, the project will enable direct experimental comparison of a wide range of available screen enlargement products used with a conventional computer monitor and used with a VCM. This approach will also support more rapid commercialization and acceptance following the expected positive experimental results.
The VCM breadboard has been demonstrated at two virtual reality conferences, where several hundred normally-sighted individuals used the system to navigate about a typical Macintosh desktop after only one sentence of instruction. While no formal data has been taken, it is clear that the interface functions intuitively, since productive interaction with the desktop folders was instantaneously achieved even among those unfamiliar with virtual reality concepts. This is supported by the observation that a very high percentage of trade show attendees navigated immediately to the ÒgamesÓ folder, where they booted ÒWolfenstein 3DÓ and asked why they could only see part of the screen.
The VCM has also been informally demonstrated to several dozen visually- impaired individuals, including several with severe impairments qualifying as legal blindness (vision worse than 20/200). In a few cases, visually-impaired users indicated that reading was still impossible, but most of the informal demonstrations resulted in positive feedback. Since a complete description of each individualÕs impairment was not recorded during these demonstrations, it is too early to predict what impairment characteristics can be overcome using the VCM as an adaptive device.
The prototype system has been made functional, but as of this date, several operational problems remain before the system can be demonstrated to visually-impaired users.
The most technically challenging problem is tracker behavior, which is unstable. At times, the gyroscopic tracker performs remarkably well, and the true promise of the VCM can be appreciated. At these times, the user can rapidly look towards any location on the virtual screen, easily position the cursor over an item to be selected, and click the mouse to operate the computer. However, the tracker is unstable, and often exhibits significant drift, hysteresis, and/or dynamic range limitations. In practice, drift shows up as a gradual shifting of the entire virtual screen across physical space, causing the user to rotate continuously to keep up with the image. Hysteresis shows up as a differential movement across the image in two opposing directions, ie; rotating 10 degrees to the right followed by 10 degrees to the left leaves the cursor somewhere other than the starting point. Dynamic range limitations show up as insensitivity to small, slow movements, and limitations on maximum rotation rate.
The developers are currently working to resolve the tracker instabilities, but it is possible that certain inherent features of gyroscopic trackers may prevent complete resolution. Should this be proven true, development will turn to a different tracking method. The most likely candidate is a combination of an electromagnetic fluxgate compass for yaw, and liquid-based tilt sensors for pitch. General Reality Company currently offers such a device commercially with an RS- 232 interface, so development would entail new interface electronics and software for connecting to the ADB port.
Outside of the tracking challenges, most of the currently planned improvements involve software. For example, it is postulated that an important feature will be the ability to maintain the cursor at any fixed position within the user's visual field instead of only at the center. This would allow users with central scotomas to use the VCM device by positioning the cursor at any preferred retinal locus. An additional requirement will be automatic calibration of tracking coefficients to match cursor slew rate to magnification setting, so that the virtual page remains stationary after magnification changes. Finally, various "clean-up" activities are required, such as eliminating the visual smearing that often occurs at the top of a VGA-to-NTSC converted display image.
Beyond these near-term improvements to the prototype required before human testing can begin, it is likely that additional required improvements will be identified by potential users. Some of the potential improvements users may demand are suggested below.
Until this point in the project, progress has been achieved informally, without major financial outlay other than time spent by the authors. However, development and evaluation beyond the concept stage will entail significant expense, so funding has been arranged in the form of a federal research grant from the Department of Health and Human Service. The grant will support three primary tasks, which are completion of the VCM prototype, visual safety testing to be performed by SRI International of Palo Alto, California, and performance evaluation to be performed by the Western Regional Blind Center at the Veterans Administration Hospital in Palo Alto.
Key goals of this effort will be to demonstrate the following advantages through performance testing using visually-impaired computer users:
During the first phase of the planned effort, the first of these projected advantages will be assessed through controlled experiments that will measure (non-forced) reading speed and computer productivity (correction of randomly placed spreadsheet entries). Using a planned 45 degree FOV version of the CyberEye, the second hypothesis will be similarly tested. The third hypothesis will be demonstrated and subjectively evaluated in the first phase of the research, and considered for detailed evaluation in a later experiment.
General Reality expects to find that in computer productivity tests, the VCM will prove more effective than fixed monitors due to improved spatial orientation and dynamic control. However, results for simple reading speed test are likely to show the VCM inferior, since continuous linear reading requires no spatial orientation, dynamic control, or other navigational capabilities. If these expectations are both proven true, it will suggest that to be useful, a commercial VCM would require a tracker-disabling mode that would support conventional continuous scrolling for continuous reading. In such a mode, the benefits of the VCM would be the larger potential FOV of the HMD compared to a conventional monitor, and portability.
General Reality plans to evaluate safety issues during this
effort with the support of SRI International, a highly respected
non-profit research and development think-tank and one of the
worldÕs leading centers of research into the physiological
effects and safety issues of HMD use. In planned SRI-controlled
tests, subjects will don an HMD supplied by General Reality that
incorporates all of the relevant HMD design variables of the
low-vision-aid system. Subjects will adjust the HMD for their
head and eyes, and they will be asked to comment on
1) the comfort of the HMD after adjustment,
2) any difficulties they had in making adjustments, and
3) whether they were able to see clear, single, fused images prior to making adjustments for the eyes,
4) whether they were able to see single, fused images after making any necessary adjustments for interpupillary distance or vertical phoria, and
5) whether they experienced any neck or body discomfort.
Subjects will also undergo routine vision tests to determine
whether the subject has:
1) developed eye strain,
2) experienced a loss in either far or near visual acuity,
3) had a change in ocular alignment or
4) a reduction in depth perception.
Routine vestibular tests of balance and locomotion will also be administered to subjects, and if possible, a simple test of muscle strain will be administered.
The application desirability issue will be addressed during the planned effort using subjective comments from expert subjects and through a focus group approach, where members of the local visually-impaired community will be invited to preview the device and discuss issues identified during the experimentation phase. To the extent allowed by the results of the safety evaluation and human subjects assurances, these volunteers will be allowed to try the device prior to the discussion.
While the long-term goal of the planned effort is a commercial product, the project team believes its initial efforts could have significant impact from a pure research standpoint for at least three reasons.
First, a large body of research currently exists in the area of reading speed, including comparisons between normally sighted and low vision subjects, study of eye saccades, and detailed evaluation of psychophysical requirements such as character size, contrast, and sampling bandwidths. However, research into corresponding issues while using a HMD is almost non-existent, and the results might be surprising.
For example, recent research measured stationary text reading rates 20% faster than scanned text reading rates and found that a four character- wide window is sufficient to maximize scanned reading speed. Compared to earlier findings of 19 characters for stationary text (15 characters to the right of fixation and 4 to the left), this work indicates different processes for these two reading regimes. If stationary text can be read faster due to improved guiding of saccadic eye movements, will the same hold true when the entire head is scanned across the text instead of just scanning with the eye? Will saccadic eye movements vary with magnification/field-of-view combinations?
Perhaps more importantly, much of the current body of low-vision aid research is not relevant to real-world interaction with computers, since the experiments have tended to study linear, continuous reading which requires no navigation. With multimedia personal computers for home use now outselling televisions in the US, it is important to measure how low vision access aids enable navigation and interactivity, not just rapid text comprehension. In fact, according to the VA Blind Center, "graphical user interfaces are a major source of concern to visually impaired computer users". To explore this issue, the planned effort is expected to develop and publish a controlled experiment that tests computer interaction productivity, not just linear reading rate.
Finally, tens of thousands of HMDs annually are beginning to see application in a variety of areas, from virtual reality games to surgical simulators. Research in the area of HMD safety is sparse, and indicates that seemingly minor errors in HMD design, manufacturing, or use can result in significant health impacts. With no regulatory body yet overseeing visual performance, setting exposure guidelines, or educating consumers, further HMD safety research under this effort might prove to provide an important "ounce of prevention" in the National Eye Institute's overall research prioritization.
The research team assembled for this project hopes its experimental approaches will stimulate further work by other researchers into data perception in HMDs, low vision computer interaction, and HMD safety. To that end, ongoing results of the research will be published as rapidly as possible. In addition, General Reality plans to release an "experimenter's kit" for use by other researchers in the near future, which will include the VCM's HMD, tracker, and software. It is hoped that commercial availability of this kit will stimulate additional research efforts in this area.
Go to previous article
Go to next article
Return to 1995 VR Table of Contents
Return to Table of Proceedings