Hold-to-talk voice input with live interim text in the composer, and a plugin API to build it #5623

Summary

I want to dictate the way most push-to-talk tools work. I hold a key, talk, and see the words show up as I speak. When I let go, the text lands in the composer. Today bb voice input is click to start, click to stop, and no text appears until the whole recording has been transcribed. I would like core to support this. Failing that, I would like plugins to get the two composer hooks needed to build it, because a local transcription plugin is the obvious place for it.

Versions and environment

• bb 0.46.0, desktop app
• macOS 26.6
• Code checked against main at 05771e3c82404328854468f0fff7f0f8203e2076

Workflow I cannot complete today

1. Open a thread and put the caret in an empty composer.
2. I want to hold one key, speak, watch the transcript build up, and release to get editable text.
3. What actually happens: I click the microphone button, speak with no text feedback, click stop, and wait for the full transcript.

Current behavior in code:

• The composer button starts recording on click (PromptBoxInternal.tsx#L4081-L4090).
• useVoiceInput records in 250 ms chunks but sends audio only once. In recorder.onstop it builds a single blob and calls onTranscribe once (useVoiceInput.ts#L435-L487, #L530).
• The voice-transcription route takes one file and returns final text only (system.ts#L672-L687).

Requested behavior

• Hold to talk. Hold Space for about 200 ms in an empty composer to start recording. Release to put the text into the draft. Escape cancels. Once the composer has text, Space types a space as usual. Space is also left alone in inputs, buttons, menus, and open dialogs. A quick tap still types a space.
• Live interim text. While recording, show the transcript so far after the caret in a muted, italic style. This preview is not part of the draft until release, so undo, mentions, and submit never see it.
• A setting to turn it off. For example "Hold Space to talk" in Settings > Voice Input.

How this relates to existing issues

• Add a keyboard shortcut for voice input #2955, "Add a keyboard shortcut for voice input" (open), asks for a rebindable shortcut that starts and stops voice input, plus showing it in the tooltip and shortcut settings. That is a toggle, so it keeps today's model of no feedback until you stop. This request is a hold gesture with press and release semantics and live text. They fit together: the Add a keyboard shortcut for voice input #2955 shortcut could be the toggle path, and hold-to-talk the press-and-hold path, both going through the same voice-input code.
• Codex voice transcription times out at 10 s for notes over ~5 minutes #5016 (closed) is the Codex service's 10 s transcription deadline. It shaped the fork patch below. Long holds are split into segments so that each request stays small. That fix raised the deadline to 65 s (provider-codex/src/ai-service.ts#L12). Segmenting still helps, because the live preview re-sends the current segment's audio about once a second.

Prototype

I have a working patch in my fork, benwaffle/bb, as the commit "Add push-to-talk with live interim transcription to the composer". It touches only the app and uses the existing transcription route:

• A 200 ms hold threshold on Space, with the focus and empty-draft rules above. Escape cancels.
• About once a second it re-transcribes the audio of the current segment through the existing voice-transcription route. It shows the result as a ProseMirror widget decoration after the caret, so the document is never changed.
• Segments rotate every 20 s. A finished segment's final transcript becomes committed text, and the next segment continues on the same microphone stream.
• A hold stops itself after 5 minutes.
• A "Hold Space to talk" switch in Settings > Voice Input, stored client-side and on by default.

I can open a PR if you want this in core.

Plugin API gap

This would work best as a plugin, for example together with a local whisper transcription service. The transcription half is already possible, because a plugin can declare an AI service with transcribe (backend-contract.ts#L1956-L1978). The composer half is not. I checked the composer surfaces in the Plugin Guide (surfaces.ts#L449-L599) and the SDK contract:

A minimal shape that would be enough, as an experimental_ addition to PluginComposerApi and ComposerCustomization:

interface PluginComposerApi {
  /** Show host-styled text after the caret that is not part of the draft. null clears it. Cleared on unmount or scope change. */
  experimental_setInterimText(text: string | null): void;
}

interface ComposerCustomization {
  /** Press-and-hold gesture on a key while the composer has focus. The host decides when the key is free (for example an empty draft with no open menu) and when a press counts as a hold. A tap keeps the key's normal behavior. */
  experimental_holdKey?: {
    key: string;
    when?: "empty-draft" | "always";
    onHoldStart(context: { composer: PluginComposerApi }): void;
    onHoldEnd(context: { composer: PluginComposerApi; cancelled: boolean }): void;
  };
}

Following the "core uses the plugin API" rule, core's own hold-to-talk would then use these same two members instead of a private decoration and key handler.

Ruled out

• Not a duplicate of Add a keyboard shortcut for voice input #2955: that issue asks for a start/stop toggle shortcut, with no hold gesture and no interim text.
• The existing composer APIs cannot do this (see the table above).
• bb voice transcribe already covers the CLI side for one-shot transcription. Live interim text is a composer-only feature, so the SDK surface above is the parity point.

Suggested priority and effort

Low priority, medium effort. Click-to-dictate is a workaround. Hold-to-talk with live feedback is mostly a quality-of-life gain for people who dictate a lot.

AGENT GENERATED