2003 Conference Proceedings

Go to previous article 
Go to next article 
Return to 2003 Table of Contents 


Paul Blenkhorn, Gareth Evans, Miltos Kritikos, Maria Schucher
Department of Computation
PO Box 88, Manchester, M60 1QD, UK
Tel: +44-161-200-3371
Fax: +44-161-200-3324
Email: p.blenkhorn@co.umist.ac.uk

Karl Shore and Andy Partington-Utterly
Sense North
Rodney Clark Resource Centre
Robin Hood, Wakefield, UK


This paper describes a software package ('Sense Factory') to support deafblind people in the development of daily living skills. The target client group are largely congenitally deafblind people who have some functional vision and/or functional hearing and who may have additional disabilities including cognitive impairments.


'Sense Factory' is designed to support the development of daily like skills including: simple food preparation (for example, making toast); personal hygiene (for example, brushing teeth); and life skills, such as dressing. The software presents a series of 'activities' that can be selected on behalf of, or by, the deafblind user. Each activity deals with a specific instance of a life skill; e.g., making toast is one activity, making a cheese sandwich is another and so on.

In order to present the features of 'Sense Factory' it perhaps best to firstly identify the sets of different users of 'Sense Factory' and consider its use from the perspective of each of these user groups. There are four potential users: deafblind users ('End Users'); 'Activators' who run 'Sense Factory' and select the activity on behalf of the user; 'Configurers' who configure the activities for the user; and 'Activity Creators' who create new activities.


The End User can be presented with an activity in two different ways, 'Step by Step' and 'What's Wrong'.

In 'Step by Step' mode the End User is presented with a sequence of 'frames' that result in the completion of a task. A frame consists of a full screen image (static or animated) and/or some associated sound, which may be spoken commentary, an appropriate sound for the activity or both. The End User moves through the sequence of frames using a single switch; when two switches are provided the user can move backward as well as forward through the sequence. A keyboard, a mouse or a touch screen may also be used for input.

In 'What's Wrong' mode the End User is presented with a sequence of frames that consist of two images, presented side-by-side, and a spoken commentary. One image corresponds to the commentary, whilst the other represents a (generally humorous) misinterpretation of the commentary. For example, when making toast, one of the steps is to "place the butter on the knife". The correct image shows the butter being scooped out of its container; the incorrect image shows the butter container balanced on the knife. The End User is has two switches, which correspond by position to the two images on the screen. If the End User selects the switch that corresponds to the correct image, he/she is presented with a reward and the next frame is presented; if the incorrect image is selected, frame that indicates an error is shown and the original frame is presented again. A touch screen, mouse or keyboard may also be used as input.

One very significant set of issues concerns the presentation of the images to the End Users who have little functional vision. There are four major considerations: the selection of objects; the provision visual cues; changes of perspective; and types of display.

It is important that the objects chosen are familiar to the user. For example, in our early work we used a toaster that was circular, rather than the more familiar rectangular shape. When a small number of End Users were asked to identify the object, the majority described the circular toaster as a plate.

A closely related issue is the provision of visual cues. End Users were presented images on which a human hand and wrist were present. They failed to identify it correctly until jewellery such as a wristwatch, rings and a bracelet were added. With these cues the identification rate rose considerably.

A further issue which is still to be resolved is the changing of perspective and the use of close up. The are two schools of thought. One states that once an object is shown and the End User has established what it is (for example a toaster) its presentation should not be varied. However, this makes it very difficult to show the detail associated with an action (for example, placing a slice of bread in the toaster). The other school states that it is permissible to vary the viewing position and to use close ups. This makes it simple to show the detail associated with an action, but may be confusing for the End User. In the current set of activities we vary detail and perspective, but this issue requires further evaluation.

The final issue associated with images is their presentation. The images generated using a high resolution digital camera, with careful control of lighting and contrast. However, experiments with a small number of End Users indicated that some, but not all, users were confused by the high level of detail. As an alternative, the images were processed by a graphics designer using photo editing software to equalise the colours for objects (for example, making a hand a uniform colour) and placing a solid black line around each object - producing the effect of hand drawn images. This improved recognition for some users, but when some colour variations were kept, to cue shadow and hence a perception of depth, recognition was further improved. A third set of images was also created, a hybrid between the photographs and the 'hand drawn' images. Equalisation of colour was performed in some areas and all objects were given borders. Some End Users found this style of presentation easier to recognise. With limited testing, there seems to be no style that suits all; so the activities that have been produced so far have three styles of image (photographic, hand drawn and mixed) and the most appropriate set of images can be selected for a given End User according to his/her preferences. One problem with this approach is the considerable amount of time that it takes a skilled graphics artist to produce the images. This issue needs further evaluation.


A very significant issue in the design of software that will be used by teachers, classroom assistants, family and other carers (we will refer to this group collectively as carers) with disabled people, is that the carers do not generally possess a high level of knowledge and skills in the use of computers. In many respects, the interface for the carers is more challenging is a more challenging design task than the end user's interface. The interface must be simple to use and must not assume a great level familiarity with the computer's operating system or general file system conventions.

The 'Activator' is the person who is responsible for running the software and getting it into a state where the End User can use it. 'Sense Factory' has a simple interface for selecting activities which is also switch accessible. This may mean that once 'Sense Factory' is running, the End User will be able to select the activity that he/she requires.

The 'Configurer' is the person who is responsible for selecting the set of activities that and the set of frames presented for a given activity to the End Use. For example, the activity that presents brushing teeth is a 38-stage activity, some frames of which may not be appropriate for a particular user. The style of image (see above) can also be selected. Sense Factory is available as both single-user and multi-user versions. In the latter version, the 'Configurer' can create groups of End Users and select the set of activities and the set of frames that are presented to each user in the group.

Finally, support is given to an 'Activity Creator' who wishes to create new activities, which may be personalised for particular users. This role should not be adopted lightly, because, as noted above, considerable work and skill is required in producing an appropriate set of images. Without expertise in this sound recording can also be problematic 'Sense Factory' provides support in terms of filing and sequencing for someone who wishes to create his/her own activities.


At the time of writing the software is not yet evaluated, but preliminary results will be presented in the session. The software described in this paper will be freely available from the web, please email the authors for details. Attendees at the session will be given a CD containing the software.


The work described in this paper has been funded by the New Opportunities Fund.

Go to previous article 
Go to next article 
Return to 2003 Table of Contents 
Return to Table of Proceedings

Reprinted with author(s) permission. Author(s) retain copyright.