Go to previous article
Go to next article
Return to 2002 Table of Contents
Roymieco Carter, Jerry Schnepp, Karen Alkoby, Mary Jo Davidson, Jacob Furst, Damien Hinkle, Glenn Lancaster, John McDonald, Lori Smallwood, Jorge Toro, Rosalee Wolfe
School of Computer Science, Telecommunications and Information
243 South Wabash Avenue Chicago, IL 60604-2301
American Sign Language (ASL) is an eloquent natural language used by over a half million members of the North American Deaf community[Ster96][Deaf00]. For access to spoken English, deaf people rely on sign language interpreters. The cost and limited availability of interpreters has contributed to isolation for many in the deaf community.
A personal digital translator that would translate English into American Sign Language, would better bridge the gulf between deaf and hearing worlds. Such a tool would provide greater privacy accessibility for the Deaf. For example, medical and legal matters could be transacted doctor-to-patient or attorney-to-client without the need for an interpreter.
While there are currently many technologies that present a digital representation of ASL, from video clips, to letter-for-letter translation of English, to still images of fingerspelling handshapes, we believe that the best approach for representing ASL lies in animated three-dimensional computer graphics (CG). The nature of conversation requires the type of flexibility that CG can provide.
ASL communication involves many parts of the body including the face, hands, torso, and arms. Producing ASL on a personal digital translator means that images of the body, not just the hands, be developed in animated form. The translator must show a significant portion of the animated signer, including detailed representation of the face. This paper describes the lessons learned in developing a first prototype and the improvements required to display ASL properly as an animation.
American Sign Language is a rich and varied natural language. Although it shares some vocabulary with English, ASL is not a word-for-word translation of English words and sentence structure. It presents many of the same challenges of any language translation process, and adds the complexity of changing modality from aural/oral to visual/gestural [Alko99a].
Word/phrase signs can express an extraordinary range of meaning by using the natural geography of the body and facial expression, in addition to the hands. As practiced by fluent signers, word/phrase signs are economical and of endless variety and account for the vast majority of a typical ASL conversation [Tenn98].
Fingerspelling uses the hands to spell out English words and numbers character-for-character. It is used for proper nouns, technical terms, acronyms, and for situations where no word/phrase sign exists. Fingerspelling slows ASL conversation, but is necessary for complete communication.
Several current technologies can present ASL digitally. They include:
Both video clip technology and still image technology are limited in their ability to create the full range of ASL that is ultimately necessary. In order to be useful in conversation, a presentation technology must be flexible enough to create new sentences from signs, taking into account such items as the correct conjugation of verbs.
We believe that computer graphics is the most appropriate technology for the presentation of ASL on a digital translator. Its support for “on the fly” creation of new animations based on both existing rules/conditions and input from outside sources provides the flexibility necessary for ASL sign translation. CG has the potential for:
Usability is a central concern of the ASL project. In concept, the personal digital translator could become a constant resource for the Deaf as they carry out day-to-day tasks. It should display signs in a clear and natural method so that oddities of the representation do not distract the user. It is important that people not waste time noticing imperfections when their priority is to use the tool to communicate. However appealing the animations might appear to a hearing person, it is imperative that any approach be tested with people who are likely to use the translator.
To test this approach, we created a three-dimensional computer model that resembled an adult female. The fact that the majority of interpreters are female was the reason for the choice of gender. The model was a jointed system of rigid components similar to a wooden marionette including a fixed, painted face. A picture of the model and a close-up of the hand are depicted in Figure 1. The hand is signing I-LOVE-YOU.
Figure 1: Original Model and Hand.
When observing the marionette-style model, people could easily recognize fingerspelling and isolated signs, but almost everyone commented on its articifial appearance. The comments focused on the hand and face. On the hand, the lack of webbing, especially between the thumb and palm, was disconcerting. One user remarked that the hand looked like it belonged to "ET." The most consistent remark about the face was that it lacked the ability to form expression and appeared "wooden."
While the issue of the hand was one of aesthetics that was distracting to users, the second issue regarding the face was far more serious because it impacted on the model's ability to convey sentences in ASL
There is consensus that the face and facial expression are essential to proper communication in ASL. Students who are learning ASL are reminded repeatedly to focus on a signer's face, not on the hands [Smit88]. Facial expressions are examples of non-manual behaviors and they express not only affect or emotion, but also have grammatical significance. Signers use these behaviors to ask questions, to make negative statements or to add emphasis. Using only the manual signs for HOME and YOU, a signer can use the appropriate nonmanuals to create the following statements:
Without facial expressions, it is not possible to form any type of interrogative statement or to express the large number of signs that include nonmanual behaviors. For these reasons, any computer system attempting to display ASL must incorporate facial expressions.
Based on use feedback, we made several significant changes to the model. We went from a marionette-type model with jointed rigid components to a model whose hands have a realistic surface the behaves more like actual human skin. Figure 2 shows the new hand making the sign I-LOVE-YOU . The webbing and knuckles now resemble a human hand more closely.
We have also incorporated facial expressions. Figure 3 shows a still frame from the WH- nonmanual, which is used to as any sort of WH- question such as "What is your name?", "Who is your teacher?" and "Where are you going?" Figure 4 shows several frames from the nonmanual "AF-FO", which is often used for emphasizing "HAVE TO".
Because the face is so important, we enlarged the head so it would be easier to see. We also made the hair shorter so that more of the face is clearly visible to viewers.
We are currently creating a definitive set of facial expressions to support the display of nonmanuals in ASL animations. The expressions are being modeled by artists with experience in life drawing and are being edited by deaf experts. As part of our presentation, we plan to show sample animations and solicit feedback on its appearance.
[Alko99a] Alkoby, K. A Survey of ASL Tenses. Proceedings of the 2nd Annual CTI Research Symposium. Chicago, Illinois, November 4, 1999. http://bach.cs.depaul.edu/ctiphd/ctirs99/online/alkoby.html
[ASL99] ASL Fingerspelling Dictionary, http://where.com/scott.net/asl/
[Deaf00] Deafworld web site, http://dww.deafworldweb.org/int/us/
[Holt94] Holt, J., "Demographic, Stanford Achievement Test – 8th Edition for Deaf and Hard of Hearing Students: Reading Comprehension Subgroup Results". http://www.gallaudet.edu/~cadsweb/sat-read.html
[Mich99] Personal Communicator CD, Michigan State University Communication Technology Laboratory, 1999.
[Sedg01] Sedgwick, E. et al., Toward the effective animation of American Sign Language. Proceedings of the 9th International Conference in Central Europe on Computer Graphics, Visualization and Interactive Digital Media. To appear.
[Smit88] Smith, C., et al., Signing Naturally. San Diego: Dawn Sign Press, 1988.
[Ster96] Sternberg, M., The American Sign Language Dictionary, Multicom, 1996. (CD ROM)
[Su98] Su, S.A., "VRML-based Representations of ASL - Fingerspelling on the World-Wide Web.
[Tomu00] Tomuro, N.,et.al, An Alternative Method for Building a Database for American Sign Language. Presented at the Technology and Persons with Disabilities Conference 2000. California State University at Northridge, Los Angeles, CA March 20-25, 2000.
[Tenn98] Tennant, R. and Brown, M. The American Sign Language Handshape Dictionary. Washington, DC: Clerc Books, 1998.
Go to previous article
Go to next article
Return to 2002 Table of Contents
Return to Table of Proceedings