Speech to text #941
Speech to text #941
Conversation
amyjko
left a comment
amyjko
left a comment
There was a problem hiding this comment.
This is very close! I'm impressed how well you picked up most of the patterns necessary to build a stream; it's one of the more complex parts of the programming language implementation.
Here are the key things to address:
- There are several error handling cases in the
Speechimplementation that either fail silently, or fail by printing to the console. All of these should be localized error values that become a new string value. SeeWebpagefor a pattern for this. Essentially, when there's an error, we want that error to show up as the stream's value on stage. (I think is is more graceful then displaying an exception, since the stream already results in a text value). - I believe your branch is out of sync with main; there are several type errors in see
npm run checkrelated to theColorJSpackage that are resolved in main, but aren't resolved here. - There are a few places where the we should and shouldn't be using the current locale selected for the UI. See comments for detail.
I will machine translate the strings prior to merging, once the error messages all have appropriate localization strings.
| // Choose the selected evaluation locale or if not selected, the user's preferred locales | ||
| evaluationLocale | ||
| ? [evaluationLocale] | ||
| : $locales.getPreferredLocales(), |
There was a problem hiding this comment.
This should not be using the locales database; projects have locales that they have embedded in them and they aren't necessarily the same as the UI locale chosen. Is there a reason this needed to be changed to use the user's current UI locale as a fallback?
| window.SpeechRecognition || window.webkitSpeechRecognition; | ||
|
|
||
| if (!SpeechRecognitionAPI) { | ||
| console.warn('Speech recognition not supported in this browser.'); |
There was a problem hiding this comment.
If the API isn't supported, the stream should return an error value as its value, rather than failing silently. See Webpage for an example of how to handle this.
| this.recognition.interimResults = false; // Only final results | ||
| this.recognition.maxAlternatives = 1; // Only best match | ||
| this.recognition.lang = this.languageCode; | ||
| console.log( |
There was a problem hiding this comment.
Drop the console messages prior to deployment.
| case 'network': | ||
| // Network error - could be no internet or service unavailable | ||
| this.react( | ||
| 'Network error: Check your internet connection and try again.', |
There was a problem hiding this comment.
All of these messages should be localized, rather than hard coded as English.
| if (this.retryCount < this.maxRetries && this.on) { | ||
| this.retryCount++; | ||
| const delay = this.retryDelay * this.retryCount; // Exponential backoff | ||
| console.log( |
There was a problem hiding this comment.
Remove this log before we finalize this PR.
| try { | ||
| this.recognition?.start(); | ||
| } catch (e) { | ||
| console.error('Failed to restart recognition:', e); |
There was a problem hiding this comment.
This should be a localized string value for the stream, not a console error. We want it to be visible on stage.
| }, delay); | ||
| } else if (this.retryCount >= this.maxRetries) { | ||
| // Max retries reached | ||
| this.react( |
There was a problem hiding this comment.
This should be a localizes string value for the stream, not a console error.
| // Get the language from the parameter, default to 'en-US' | ||
| const languageCode = | ||
| evaluation.get(languageBind.names, TextValue)?.text ?? | ||
| 'en-US'; |
There was a problem hiding this comment.
I think this should default to the preferred locale in the locales argument passed to createSpeechDefinition. That way, if the creator has Spanish chosen for their UI locale, the default will be Spanish, rather than English. Call locales.getLocale() as the default.
| "Note: Speech recognition requires microphone access and an internet connection in most browsers. Also, there might be usage limits, so don't leave it on for too long!" | ||
| ], | ||
| "names": ["🗣️", "Speech", "Voice"], | ||
| "reset": { |
There was a problem hiding this comment.
Need to run npm run schemas to update the schema to allow these input keys.
| "Speech": { | ||
| "doc": ["$?"], | ||
| "names": ["$?"], | ||
| "reset": { |
There was a problem hiding this comment.
I will translate these before merging.
2 participants
Context
I am adding a new speech-to-text Input Stream called
Speech, using the Web Speech API.This will enable voice input and appeal to those who can speak a language but are still in the process of learning how to read or write.
Wordplay users can configure the stream through:
reset: Clears accumulated speech when set to truelanguage: BCP-47 language code (e.g., en-US), defaults to en-USlimit: Max number of words to keep, defaults to infiniteRelated issues
N/A, this is a new feature. With that said, the localization team will need to translate the Speech Stream descriptions.
Verification
I tested on both Chrome and Safari, and both work as intended. The testing was primarily done in English, with some additional testing in Chinese, Japanese, Korean, and Spanish.
Checklist
N/A, we are ready to ship! 💯