1995 VR Conference Proceedings

Go to previous article 
Go to next article 
Return to 1995 VR Table of Contents 


MULTIMEDIA AND VIRTUAL REALITY TECHNOLOGY: MEASUREMENT AND RECORDING IMPLICATIONS FOR HEALTH CARE

Presented By David Udy, Dr. P.H.
California State University, Long Beach
8251 Tula St., Long Beach, Ca., 90808
(310) 493-68977

ASTRACT

With the emergence of revolutionary new multimedia and Virtual Reality (VR) technologies, it becomes necessary to reconsider how health care is both measured and recorded. At the current period of development, the focus on health related applications of VR technology has attempted to define it's "use", rather than to contemplate the long term "effect" of the new VR culture on the health care delivery industry. This paper begins with a definition of multimedia and VR concepts. Following this introduction of the technology, an examination of the changing world of health care delivery is offered from three different perspectives: 1) size & space, 2) motion & activity, and 3) time and duration. In addition, the issue of an "Intelligent Multimedia Medical Record" is discussed.

DEFINING VIRTUAL REALITY AND MULTIMEDIA

It can be said that the first attempts at VR were the earliest storytellers and tribal historians of ancient cultures who attempted to transport their audience to times and places only imagined.. The storytellers and historians of old may have been replaced by digital technicians and 3-D animators; and the communal experience of sitting around the campfire is giving way to the shared virtual experience of a "goggle headed" audience. Throughout this technical revolution however, a key cognitive concept that has survived is the notion of, "suspension of disbelief". As the tools for creating these "virtual worlds" have evolved from earliest hieroglyphics, to printed material and books, to electronic media of; radio, moving pictures, television, and on to today's state of the art 3-D rendering systems, the objective of creating a better representation of "real multisensory experiences" requiring less "suspension of disbelief" remains a driving force for future research and conceptual development in multimedia and virtual reality. As noted above, the conceptual foundation of VR is not new, but it is the emergence of creative and powerful new technology that require the reconsideration of the conceptual foundation of health care delivery itself. From molecular engineering using micromanipulated laser scalpel, to a virtual cardiothoracic surgical workstation, to four dimensional cardiac system imaging, etc., revolutionary new technology is forcing the health care industry to reconsider the meaning and context of such bedrock concepts as, time, space and motion. When one considers something like the emerging Magnetic Source Imaging (MSI) technology used to identify navigation pathways for neurosurgeons preoperatively, it becomes evident that we are at a pivotal point in human development where "virtual worlds" no longer require "suspension of disbelief", but are actually perceived as superior and preferred over every day life sensory experience. It should be noted that the potential risk and damage from this evolution is significant and requires extensive research but due to space limitations these subjects are largely beyond the scope of this overview paper.

As traditional libraries move away from "stacks of books" to "stacks of bits" it seems as though the digital snow ball approaches critical mass. While multi media technology is extremely powerful at capturing, storing, reproducing, etc., all types of digital knowledge bits, the technology remains a traditional file based system that required the user to navigate outside the digital domain. With VR technology however, the 3-D environments are created "on the fly" in a "real time" rendering process that allows for the user\actor to actually take part in the composition of the computer experience rather then just calling on preprogrammed digital elements (files), as is the case with multi media systems. While the development of 3-D technology and environments is not new, it is only recently that the cost of "garage" VR systems have become feasible. One way of classifying systems is to consider a VR model of encumbrance and immersion. The first consideration is the level of physical encumbrance used by each system. This can range from fully encumbered systems (i.e. body suits), to goggles or gloves, to the current state of the art which are unencumbered systems, where the user\actor\patient\client moves into a "virtual space" unattached to any physical apparatus or device. The second variable in the model is the level of immersion into the VR environment. This may range from something like the remote manipulation of simulated objects to a optically aided microsurgery, or a virtual tour of the surgical site preoperatively. As multimedia and VR technology moves forward at break neck speed it is impossible to predict what the future may hold in this new cyberspace. The one thing that does seem to be a cornerstone of the emerging technologies is the increasing movement towards 3-D graphics for both multimedia and VR representations.

As the entire spectrum of digital communications continues to change, the single defining characteristic that currently separates multimedia and VR environments is that VR systems generate objects, environments, images, etc., in "real-time" compared to multimedia which is largely "stored" material. Multimedia environments are constructed using a file system that includes animation, audio, video, etc. files that have been stored in essentially a completed form. VR environments may use similar objects but they would only be generated (rendered!) if needed or requested by the user\actor. This dramatically minimizes the physical size of VR environments over multimedia systems in two ways: 1) only the instructions need to reproduce objects is stored compared to the large finished object file as with multimedia, and 2) only the "needed" files are ever rendered Vs large multimedia files that are stored fully rendered. Additionally, the VR systems will require significant computer power and some "rendering engine" to generated objects as needed, while the multimedia system will need minimal computer power and only a 'small" application to manage files and handle the presentation of the stored material.

It would seem that the future will likely lead to some hybrid system that will utilize both stored multimedia files for some components and conduct "real-time" rendering for other pieces of the application. The pivotal point in the development of computerized 3-D environments into full VR systems comes, not in the definition of the technology, but in the ability of the technology to "immerse" the user in that environment. The transition from external user control to internal user\actor control marks a moderate change in the technology but a profound change in the experience felt by the user\actor. Currently the technology producing some form of VR experience range from environments that use some form of mechanical tactile interface such as power glove, glasses, head mounted displays, etc., to environments or simulations that respond to sensory environment that envelopes motion, light, sound, heat, etc., to provide an "unassisted" VR experience for the user\actor. The emerging technologies, both the hardware and software, require a redefinition of computer interactively. Below is a list of concepts that generally characterize current 3-D and VR development:

* Reconceptualization and measurement of health in 4 dimensions: 3 spatial and 1 temporal

* Defining gravity and relative size in simulated environments

* Accounting for inertia, mass, force, etc., of objects in 3-D environments

* Defining objects characteristics, such as bounce, elasticity, rebound, velocity, etc.

* Determining appearance, lighting, brightness, reflection, texture, etc.

* Recognizing motion such as: angular motion, inter object trajectory, navigation,etc.

* Identifying location of an object in 3-D space using at least 6 degrees of freedom: position coordinates in space of X, Y, Z and directional angles of Yaw, Pitch and Roll.

* Detect, record and respond to collisions

Even the line between real-time rendering of VR environments verses the stored file nature of multimedia is being blurred in the area of digital video compression. This particular hardware\software arena has exploded recently with compression programs that capture and store video or other images in a "virtual" form in which there are only computer algorithms that must be run to construct an image when a "real" image does not exist. The area of health science bio-statistics is another example of this concept when one considers various data sets and analysis that a statistition may do in cyberspace but arguably "do not actually exist" until a file is save or hard copy print is produced. This discussion points to the heart of the problem of managing clinical or administrative functions in an environment that is so dramatically different with each coming day. Health care is a industry that is struggling under the weight of largely text and numbers alone. Now that we have broken the threshold where "virtual" is superior to "real" we have available a wealth of instrument data, camera perspectives, sounds, images, etc., that can do a far better job of representing "health" then can existing systems. Only a few years ago it seemed we had a industry that was searching for technology to solve problems, now we have technology crying out for problems, but the human component is struggling with speed of that conversion. Below are several issues that have surfaced which may help define the future:

1) A critical mass of computer power has reached the desktop\garage to prompt an exponential growth in research and development in the areas of multimedia and VR internationally

2) The exponential growth of digital communications, from local area networks to the worldwide Internet, contributes significantly to the distribution and assimilation of all emerging technologies

3) Increasing bandwidth available for data communications through improved hardware such as fiberoptics combined with digital compression systems has made the issue of moving quickly on the information highway an economically solvable problem in a remarkably short time

4) The concept of a Virtual Corporation" is taking shape in which the most important contributions made to a organization come from, or are significantly influenced by, remote digital locations.

In addition to traditional administrative functions, clinical functions such as:

1) remote control of instruments and apparatus, 2) automatic sensory input from an increasing array of organ systems, machinery, environments, etc., 3) remote video feeds from the house, clinic, physician office, etc., 4) audio signal from monitors, microphones, recorders, etc., are inevitable.

5) The emergence of multi media and VR technology represents a pivotal point in the development of health care informatics in which, instead of problems looking for a "technical fix", the technology is now becoming so adaptable and powerful that it is "in search of problems"

6) There is little question that health care technology will continue to increase in both power and application at a "revolutionary pace" for the foreseeable future. Correspondingly, human understanding and capacity to assimilate, seems destined to continue at a painfully slow "evolutionary pace". As a result, the health care industry is woefully behind the race and loosing "technical" ground daily due to the inability to fully utilize emerging technology The paradox of VR is that, the objective is to develop reusable, generalized, virtual experiences but VR remains a very "personal experience" because the user\actor and their "personality" ultimately becomes an object in some form of 3-D computer simulation or program. This allows the user\actor to interact with the computer system(s) in a very individual and personal way. It is fundamentally the immersion of the user\actor "inside" the computer environment that is so different from traditional computer experiences. The nature of traditional computer interfaces such as the keypads, pointing devices, touch screens, etc., has the user in a position "outside" of most computer environments. In VR, the user\actor becomes a dynamic part of the virtual environment\experiance rather then someone outside the system using traditional methods to communicate with the computer code and graphical user interfaces (GUI's).

It is the absolutely revolutionary perspective of being there, "inside", a computer simulation or environment that makes VR such a completely personal experience. A experiences evaluated or assessed from all of the many senses complex humans evaluate any experiences in there life. While the material accessed by different people in, for example, a "VR museum" environment may be similar, the actual "experience" had by each of the VR participants is variable and personal. This would compare to a stroll through a "real" museum in which the same participants would have a similar presentation of material but would walk away with wildly different recollections and evaluation of both the experience and the technology. It is this "variable virtual experience" that brings us to the point of redefining health itself.

Redefining Size and Space: The potential impact that VR environments have on perception, and the documentation of size and space, as it pertains to health care, is profound. The blending of humans and computers in the 3-D world allows for computer based measurement and recording of human experience in significantly more discrete increments and detail than ever before. The digital domain of VR allows users to simply determine what measurement increment is the preference to use, such as inches, centimeters, pixels,, twips, etc.. With the selection of a aspect ratio for magnification, the environment can be presented in a microscopic, molecular, organ system etc. perspective. In the digital world, a profoundly rich range of options, views, perspectives, etc., are available with the setting of a few preliminary application preferences to configure the view to the users liking.

The challenge of tomorrow is to make use of the technology that is emerging today. This ability to allow the user\actor to experience the material from a boundless variety of sizes, perspectives, angles, directions, etc., without any of the encumbrances of the physical world presents several special challenges for health care. Consider the issues of size and space as they are conceptualized and measured currently compared to the emerging examples below.

Example #1: Biotechnology Traditional methods for determining the structure of biomolecules is merging with VR and allowing researchers to immerse themselves inside of actual molecular structures. With force-feedback micromanipulators and microactuators the researchers are actually able to study the bonding of the atoms within sophisticated molecular structures by physically touching, grabbing, jiggling, etc., individual or combinations of atoms. Desktop DNA is on the way and ultramicrosurgical techniques will open new domains such as remote micro operation, organ operations, and molecular treatment in a cell!

Example #2: Virtual Mental Health A large hospital network has had significant breakthroughs with patients suffering from acrophobia. A VR system allows patients to experience various sensations of height until they are comfortable with a 12 story trip in a "real" glass elevator. It appears that reality is simply "too real" for these patients and this technology offers a step before reality. While they see this as a computer-generated environment but also as a place where they can confront and work through their fears. The researchers plan to add VR modules for claustrophobia and agoraphobia in the near future.

Example #3: Microsurgery Such tools as the laparoscope, laser scalpel, and interchangeable tips for micromanipulators have changed the style of surgery forever. In the future most operations will happen inside of the body working in a microscopic world. A miniaturized electronic sector scanning ultrasonic detector has been developed to differentiate string-like tissue into choleduct, artery, vein, nerve, or connective tissue instead of using human fingers. Micronavigation systems including positioning sensors, microgyroscope, microaccelerometer, immuno-fluorescent optical fibers, etc., will empower health workers in ways never dreamed of. In summary, it can be seen from the examples above, that health care personnel will be hard pressed to broaden their definition of size and space as fast as those definitions are broadening themselves.

IN addition to the training challenges in the future, the necessity to maintain expanding systems of calibration, data flow, documentation, etc., remain solvable but elusive problems. Redefining Motion and Activity If virtual technology is used to determine or even conduct a medical intervention such as a cardiac or eye surgery, how are the other segments of the system outside of the surgery team, such as medical records, quality assurance, business office, readmissions etc., to also have access to the experiences of the surgical procedure? Currently, physical therapists note traditional range of motion measurements on paper documents at the end of a therapy session. In the future, four dimensional modeling will automatically identify a patients exact location within 3-D space and at the exact instant in which a stimulus or encouragement is introduced.

Example #1: Virtual Physical Therapy A creative VR prototype system allows patients to move "unencumbered" through out various virtual environments that stimulate physical movement. Output measurement goes beyond simple range of motion to include speed, direction, repetitions, duration, etc., which are all related to the instant the various virtual stimulus are presented in the environment. Researchers are exploring a "Hydro VR" system to allow people with disabilities to assess virtual environments in rehabilitation pools and aquatic settings.

Example #2: Surgical Navigation The human pelvis is a complex three dimensional figure that, when fractured, can present a significant problem for reconstructive surgery. In this operation, the method, pathway, and process of reconstruction are determined preoperatively based on x-rays, CT and MRI sequential scans that combine to provide a stereoscopic display of the true 3-D space prior to any incision being made. The surgeon is guided to a pathway and the surgical activities are based on the virtual reconstruction.

Example # 3: Virtual Organ Systems The virtual heart and a simulation of the immune system are examples of tools that will revolutionize research, training, treatment, in the near future. Just as airplane simulators are evaluated in a wind tunnel, so to will the virtual body be subjected to changing simulations to evaluate blood flow, disease progression, immune deficiency analysis, etc., in creative new ways.

In summary, an industry that measures movement in rather static terms is ill equip to convert to 3-D spatial measurement in a real time environment. A constant question for the acquisition of any new equipment should be: " What type of output is produced and how is it managed?". Redefining Time and Duration The ability to expand or contract time is another artifact of entering a digital world. In addition to "controlling" time, it is now possible to identify the instant that any health related phenomena occurs. This is not recording the hour the therapy took place but the instant that the stimulus was introduced.

Example #1: Real Time Physical Imaging High resolution ultrasound imaging is proving superior to slab or rotary CT & MRI scanning techniques because it eliminates the time delays in scanning and produces a real time representations of the heart under stress. It is possible to watch the physical characteristics of the blood as it circulates throughout the heart in a real-time fashion and over time or varying degrees of stress.

Example #2: Electronic Lab Animals Various projects are underway to develop simulations of lab animal systems to create a new generation of tools for biochemical research. These simulations allow for research that is too time consuming for traditional animal studies. Simulations to monitor white blood cell populations with the introduction of various antibodies or chemical agents are examples of research that would take years in traditional animal studies but are quite manageable in the digital world.

Example #3: Surgical Simulators Emerging surgical simulators and virtual cadavers allow health personnel to work on their craft for as long and as many repetitions as needed to practice a skill. Operations can be practiced prior to any actual surgery. New techniques can be learned and old procedures practiced in a risk free environment and without the burden or imposition of time. Medical processes can be sped up or slowed down, depending on the wishes of the user.

VR MEDICAL RECORD

From any current review of the "state of the art" in health systems, it is apparent that multimedia and VR are emerging as critical elements in health care delivery. From clinical functions to administrative support, technology is becoming more pervasive in every aspect of the health care industry. While most health care personnel are grudgingly giving way to the idea that the "art" of medicine, is in many ways, becoming the "science" of information. Considering the time and effort it has taken to move the industry to the point of "considering" a digital (text) medical record, one must wonder how long it will take to develop a true multimedia medical record. The pivotal remaining question is: "As these revolutionary new tools begin to reshape the future of health care delivery, will the evolutionary pace of the workforce be able to fully understand, utilize, or even appreciate the power that is at hand?". The current explosion in medical record vendors who tout their ability to compress the most individual patient records on to a single CD or gigabit of space seems misguided. As we move into the age of managed care and multimedia, the incentive is to capture the most possible available information about a single patient on a single disk, and not the current approach of archiving the least necessary information about the most possible patients on a single disk. The rapid drop in cost and then increased availability of tape and disk media for storage and transport of the medical record of the future is "off the shelf" equipment at local computer dealers or mail order catalogs. It can be expected that as more providers recognize the need to move into a multimedia format, vendors will adjust their services to include the needed areas of knowledge acquisition, knowledge engineering, staff training on emerging technologies and concepts, digital quality assurance, cross record analysis, etc., which are all areas foreign to a "paper" record but are at the heart of making full use of a multimedia medical record. The primary power of digitizing the various components of a medical record and having everything in the same environment is the great flexibility that then exists to allow users of all types to access and benefit from the material in dynamic and creative new ways. Below are several considerations regarding the development of a new medical record for the future:

#1 Medical Record Design

It is easy to conceptualize the medical record of the future as a single CD per patient with all the records, images, sounds, etc., compiled in a single place. The disk would be populated from instrument data in the laboratory (or home!), MRI 3-D renderings from the imaging department, segments from VR exercises in the rehab unit, etc., and have everything updated automatically. Much of the information that is standard in the medical record such as routine labs, vital statistics, nutrition, etc., could be linked across the institution to monitor treatment by category, control infections, develop treatment pathways, etc., for both real time and retrospective activities. Patients would be able to have a self contained record and providers (both internal and external) would be able to access, evaluate, and download any or all of a record, on- line. The institution would maintain normative values, establish neuro networks control groups, reference ranges, etc., as a local rest stop on the electronic highway. Each stop would essentially be linked to all other providers internationally. This would provide a different epidemiological view of the planet and facilitate the sharing of health knowledge in a different "one cultural" fashion. The rapid development of telephony technology is quickly overpowering many of the traditional problems of bandwidth & transmission speeds, protocol standardization, user access, etc., that have precluded the health care industry from aggressive debate about what comes after, the long sought "Automated Medical Record version 1.0". The challenge of representing and transmitting 3-D images, VR simulations, digital video, etc., in a true "Multimedia Medical Record" is now a solvable problem. Addressing this problem begins with the establishment of common standards for cross platform exchange of files and media. Application developers must continue to include extensive export and import capabilities to facilitate movement of "knowledge bits" across the planet regardless of which operating system, hardware platform, authoring tool, development environment, etc., is involved at any given stage of the process. Neuro network technologies can be used to investigate the un-edited data bases in "data mining" sessions in which the software persistently moves freely among the data looking for new and different associations between the various technologies and human objects in the system. As the technologies change, the neuro-net maintains a constant vigilant search through the growing local and remote data files looking for associations the traditional health care researchers would never find without VR environments and measurement. VR environment and application development will be driven by creativity and revelations from the data driven perspective of neuro networks, expert systems, fuzzy logic, etc., to bring VR into the health care delivery system mainstream. Use of data to build programmable body system simulations to allow for professionals to view various measures of system involvement or impairment in "realistic" VR environments. The ability to present a "data driven" simulation of the human body in a 3-D environment in which size and space for both the "virtual cadaver" and the user\actor are computer controlled will change our cultures perception of our bodies and this species forever.

#2 Record Format & Content

It is in the areas of "format and content" in which the creative future of multimedia records is to be written. It is quite easy to conceptualize a record which contains a video from a history & physical, a 3-D image of the brain, or a lab record in hypertext format, but the dissemination and assimilation of the powerful new technologies of expert systems, VR, neuro networks, fuzzy logic, etc., requires a larger leap of faith by today's, generally computer illiterate health care workforce. The first question of record format is: "What material in the record is to be multimedia (i.e. stored in an essentially final format) and what material is to be "virtual" (i.e. rendered or produced as needed) ?". Basically, the current concept of a personal medical record is a document that contains only "data" directly related to the individual. In a emerging digital world, some "personal applications" could be placed on a individuals disk. These small neuro networks, forms processors, expert systems, personal monitors, etc., could monitor insurance benefit use, blood chemistry, nutrition, pharmacology, exercise, etc., at the lowest and the most important level, the patient or client. This "digital medical applet" approach suggest that the reengineering of the medical record away from the traditional clinical department format that facilitated paper filing technology, and towards a "body organ or system level" format could assist in organizing the new "interactive" multimedia record format. All information about various clinical and administrative systems of the body could then easily be found in the same area regardless of what type of file format the material was in. Confounding the critical issue of "format" is the seminal issue of "content" which is at the heat of the first part of this paper. It appears that the emerging technology of the digital world will allow for the acquisition, storage, and retrieval of a seemingly endless and growing amount of data, information, and knowledge about any given patient or client. Thus the question becomes: "If we could include anything we wanted in our record, what would it be?". In today's paper world a surgical procedure is reflected in the records by a few lab reports, pathology report, physician dictation, etc., giving a vision far removed from what actually takes place. In the emerging digital world it is possible to capture growing amounts and types of instrument data, handwriting and voice recognition inputs, laparoscopic and other video, ultrasound, VR, imaging signals, etc., which could be critical elements in quality assurance, utilization monitoring, treatment pathways, training, and which may also have great relevance in the "personal health knowledge base" of the future. An surgical operation or physical therapy session of today is viewed in terms of gross success or failure of various general activities that took place during the period. In the future, the perspective becomes one in which numerous instruments, cameras, apparatus, etc., produce a massive amount of material which creates a "virtual" representation of what actually took place. To bridge the gap between today's record, which seems to contain "too little" information and tomorrow's record which threatens to contain "to much" information, there must be the development of fuzzy logic, expert, and neuro networks, to construct a framework of an "Intelligent Record" that maintains, monitors and continually evaluates a growing number of personal health related data, information, and knowledge bits. This evolution could lead to the use of the patient or client record as a tool for international knowledge exchange.

#3 Record Support Documentation & Specifications

The growing complexity of health services require a significant amount of supporting information in the way of documentation and equipment specifications. Such things as file formats, calibration data, reference ranges, equipment used, color pallets, speed and movement characteristics, event sequence, etc., all need to be included to make the record a dynamic and creative multimedia platform for redefining personal health. This would include any training needed to operate and navigate the record as well as electronic links to local, regional, national, and international resources as needed. Constructed using an object oriented programming framework, the record should be standardized in its format in that it is simply a "point and shoot" environment to facilitate transferability and usability. This structure essentially bypasses the often misguided and slow national initiates at "data standardization" with a focus on "system standardization". Standardized transfer protocols allow local facilities and physicians to include what ever they feel is appropriate in each record, by following only a few "technical not content" rules for data acquisition and submission to the record. Instructions on how to calibrate a set of VR goggles for the recreation of a physical therapy simulation or the identification of pathways to reference an existing patient video history and physical for a new therapist, could all be included in the record documentation.

#4 Security & Quality Assurance

Who can review? Who can submit? Who can download? Who can modify? These are only a few of the provocative new questions that surface with the emerging technology. Digital tumors could be erased, digital heart murmurs could be silenced, digital records could be downloaded to the Internet in an instant. The issues of data security and health care quality assurance have historically been different domains, in the emerging digital world these activities become more closely related. Consider the following scenario, you are floating down the virtual artery system and a lesion appears that the physician doesn't see, or worse yet, actually created with the laser scalpel, is it malpractice? How is the record protected so the lesion doesn't disappear or the file get deleted? As a father of a "cyberteen", it seems that extensive effort spent at keeping people out of records is misguided. An alternative focus of developing systems to track and penalize unauthorized access or use seems to be a more practical approach then to perpetually challenge the "hackers of tomorrow".

Go to previous article 
Go to next article 
Return to 1995 VR Table of Contents 
Return to Table of Proceedings 


Reprinted with author(s) permission. Author(s) retain copyright.