Design Principles of an Automatic Speech Recognition Functionality in a User-centric Signed and Spoken Language Translation System

Abstract

The European project SignON aims at designing a user-oriented and community-driven platform for communication among deaf, hard of hearing, and hearing individuals in both sign language and spoken languages (i.e. English, Dutch, Spanish, and Irish). Inclusion, easy access to translation services and the use of state-of-the-art Artificial Intelligence (AI) are the key aspects of the platform design. Users can communicate to the system with text via typing, speech via a microphone, and sign language through video, while the system can respond using, for instance, (translated) output, subtitles, translated audio via speech synthesis, and via a 3D avatar. In this framework, the design of a flexible, user-friendly component for Automatic Speech Recognition (ASR) is a challenge, due to the constraints imposed by the platform, in terms of usability and the flexible use of system-external services. This paper addresses the current state-of-the-art ASR component in SignON and the conceptual choices underlying the design, operation, and integration of the ASR component
in the SignON application.