SpeechRestore – Assistive Technology for the Speech Impaired
The SpeechRestore speech recognition application on the user’s smartphone / tablet recognizes the speech impaired user’s words and then plays the words clearly articulated through the smartphone’s speaker. SpeechRestore offers the option of either speaking the keyword or touching a button on the screen to play each word or phrase.
No costly equipment is required – SpeechRestore is an app that is easily loaded on the user’s smartphone or tablet
No connection to the “cloud” is required, so that users do not need mobile data or Wi-Fi network connection for SpeechRestore to work.
A Proof of Concept SpeechRestore application has been developed for a user with Primary Lateral Sclerosis (PLS), a type of ALS that has severely affected our user’s ability to speak, diagnosed as dysarthria (unclear articulation of speech). Our user is using SpeechRestore to help him communicate.
Here is how it works: The SpeechRestore speech recognition application on the Android smartphone / tablet recognizes our user’s words and then plays the word or phrase through the smartphone’s speaker, with a voice like our user’s voice before his illness. The application offers the option of either speaking the keyword or touching a button on the screen to play each phrase.
Initially state of the art speech recognizers were tested, including Amazon Echo, Dragon NaturallySpeaking, and Google Voice. They recognized none of our user’s words, a word error rate (WER) of 100%. These packages have acoustic models created by samples of many thousands of people’s speech. However, these samples do not include the voices of speech impaired persons such as our user. These recognizers do not expose APIs to allow the creation of new acoustic models or limiting the dictionary.
A recognition engine was selected that allowed adapting and creating new acoustic models, limiting the dictionary, and optimizing the design. Also, the decoder engine was required to run locally on smartphones (Android and iOS) with acceptable performance. With the decoder running locally, the user does not have to be connected to mobile data or a Wi-Fi network (Amazon, Google, and Apple recognizers all require connection to cloud computing).
A custom acoustic model for our user was created on Linux using recordings of our user’s words and the recognition engine was optimized for minimum WER. This model and the decoding application were moved to the Android smartphone / tablet platform.
SpeechRestore uses the recognition of a word to play a word or phrase. For example, recognizing the keyword “water” plays “I need some water” and for “help”, “I need help!”. To provide a convenient list of the phrases with the keyword for each phrase and to provide an alternative method of selecting a phrase, buttons with the phrase name and keyword are provided on the smartphone / tablet screen.
The user has the option of speaking the keyword or touching the button to play a phrase. There are four buttons per row, with the phrases arranged by rows, color coded by these categories: help, actions, questions, greetings, positive responses, negative responses, ending a conversation, and information. Using the screen scrolling feature in Android, there is no limit to the number of phrases – currently there are 65. The application adjusts automatically for screen size, so it can be used on both tablets and smartphones. Our user is using the application to help him communicate.