• 39th CSUN Assistive Technology Conference - March 18-22, 2024 at the Anaheim Marriott

    The Premier AT Conference

39th Annual CSUN Assistive Technology Conference Has Concluded

LLM-Based Automatic Sentence Generation for AAC Devices

Date & Time
Wednesday, March 20, 2024 - 2:20 PM PDT  
Elite 1-3  

This work presents a system for the automatic generation of complete sentences using large language models (LLMs) based on the selected combinations of symbols for AAC devices. It first uses speech-to-text (STT) to transcribe spoken utterances to texts for extracting the context of the given situation, then it takes input in terms of selection of symbols from the AAC user. The contextual information as well as the selected symbols are then used as inputs to an LLM, GPT-4 in our case, to generate a complete sentence suitable for AAC-based communications. Then a text-to-speech (TTS) plays the speech using the generated sentence. For enabling natural communication, the system needs to guarantee real-time operation by using LLM, STT, and TTS application programming interfaces (APIs) in a sequential manner, and careful prompting techniques to generate proper sentences suitable for the context of AAC device users. In addition to these, this work presents methods to dynamically change the layout of the symbols based on the context learned from the STT results as well as previous conversations, for efficiently utilizing the limited screen real estate. It also presents the results of user studies for people with communication disorder, autism spectrum disorder (ASD), and brain lesions.

This Presentation Link is provided by the Presenter(s) and not hosted by the Center on Disabilities at CSUN. The Center on Disabilities has confirmed, as of March 27, 2024, content linked is relevant to the presentation, but has not been reviewed for accessibility nor will the Center on Disabilities attempt to remediate any accessibility issues in the linked content. Please contact the Presenter(s) with any accessibility concerns.

  • Higher Education
  • Information & Communications Technology
  • Disability Specific
  • K-12 Education
  • Research & Development
Audience Level
Session Summary (Abstract)
The presentation will describe the system integration of LLM, STT, and TTS for enabling natural communication for AAC devices by exploiting the capabilities of LLM for generating complete sentences given proper prompting.  
Primary Topic
Augmentative and Alternative Communications (AAC)  
Secondary Topics
  • Artificial Intelligence (AI) & Machine Learning (ML)
  • Emerging Technologies
  • Engineering
Session Type
General Track  


  • Bowon Lee
    Inha University/SpeechWise Inc.
  • Jong Hwan Na
    Inha University/SpeechWise Inc.

Add to Outlook/iCal (How it works)

Back to Session List