Skip to content

feat: record PC audio during online meetings - #20

Merged
piyton merged 6 commits into
mainfrom
feat/online-meeting-system-audio
Aug 28, 2026
Merged

feat: record PC audio during online meetings#20
piyton merged 6 commits into
mainfrom
feat/online-meeting-system-audio

Conversation

@piyton

@piyton piyton commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Closes #18.

The problem

Recording a meeting captured the default microphone and nothing else, so a Teams,
Zoom or Meet call produced a transcript containing only the local user. The
workaround was recording the call in OBS Studio first and transcribing the file
afterwards, which makes the built-in recorder pointless.

Two smaller defects sat in the same code path:

  • settings.audio_device was written by the Settings UI and stored, but audio.rs
    never read it — it always opened host.default_input_device(). The dropdown did
    nothing.
  • There was no way to tell whether meeting audio was arriving until the recording
    was over.

What changed

Record & Transcribe now asks whether the meeting is in-person (microphone
only) or online (microphone plus PC audio). Online opens a second cpal stream on
the render endpoint, which cpal turns into a WASAPI loopback client — it sets
AUDCLNT_STREAMFLAGS_LOOPBACK itself, so no new dependency was needed.

Dictation via the global hotkey stays microphone-only even while a meeting is
running, so a dictated sentence can never pick up whatever the machine is playing.

Why a timeline mixer

Appending samples as they arrive does not survive system audio. WASAPI delivers no
packets at all while nothing is playing, so every silent stretch would shrink and
the system source would slide ever further ahead of the microphone — badly out of
sync well before an hour-long meeting ends, and overlapping speech is exactly what
Whisper handles worst.

OBS Studio solves this in its mixer rather than its capture, and this PR does the
same. TimelineMixer (new, src-tauri/src/mixer.rs) addresses one shared 16 kHz
timeline by capture timestamp: gaps become silence, overlapping sources are summed
and clamped. Timestamps come from InputCallbackInfo::timestamp().capture, which
on WASAPI is an absolute QPC value, so both streams share one clock. The first
microphone packet anchors position zero. take() leaves a 300 ms tail so a packet
from the slower stream still lands in place when the recorder pulls a segment.

Also in this PR

  • settings.audio_device is finally honoured, for meetings and dictation alike.
  • New settings.system_audio_device plus a Settings → Audio → System audio
    device
    dropdown, for switching between headphones and speakers.
  • A PC level meter beside the microphone meter, and a warning when no PC audio has
    arrived for 30 seconds — the failure you would otherwise only discover afterwards.
  • A configured device that has been unplugged falls back to the platform default
    instead of failing the recording.

Failure handling

A meeting should never be lost to an audio error.

Situation Behaviour
Loopback fails to open (macOS, Linux, no output device) Logged; recording continues microphone-only and the UI says system audio is unavailable
Output device disappears mid-recording Logged by cpal's error callback; the microphone keeps recording
No system packets for 30 s in online mode Warning in the panel; recording continues

Testing

16 unit tests, all passing (cargo test --lib): the mixer's gap-to-silence,
out-of-order writes, summing and clamping, the drain margin, stale-write rejection;
plus device resolution and the downmix/resample helper.

One opt-in hardware test verifies the real thing on Windows — it renders a quiet
440 Hz tone on the default output and asserts the loopback stream picks it up and
reaches the mixed timeline:

cargo test --lib -- --ignored loopback

Verified passing on Windows 11. npm run build and tsc --noEmit are clean.

Not in this PR

Keeping microphone and system audio as separate tracks for speaker attribution
("me" versus "the others"). Attractive for minutes, but it doubles transcription
time and deserves its own design.

Design and plan: docs/superpowers/specs/2026-08-28-system-audio-capture-design.md,
docs/superpowers/plans/2026-08-28-system-audio-capture.md.

🤖 Generated with Claude Code

piyton and others added 6 commits August 28, 2026 12:32
Covers issue #18: the meeting recorder captures the microphone only, so
remote participants never reach the transcript.

Design mirrors OBS Studio's audio mixer: WASAPI loopback capture (already
supported by cpal 0.15.3) plus a timestamp-based timeline mixer, so silence
on the system source becomes silence on the timeline instead of drift.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mixing two capture streams by arrival order breaks as soon as system
audio joins: WASAPI loopback delivers no packets while nothing plays, so
every silence would shrink and the system source would drift ahead of the
microphone. Address the timeline by capture timestamp instead, the way
OBS Studio's audio mixer does, so a gap becomes silence and stays put.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An online meeting now opens a second cpal stream on the render endpoint,
which cpal turns into a WASAPI loopback client, and both streams mix onto
the shared timeline. Remote participants finally reach the transcript.

Also honours settings.audio_device, which the Settings UI has been writing
since forever while audio.rs always opened the default microphone, and adds
settings.system_audio_device for picking which output to capture.

Dictation stays microphone-only even while a meeting runs, so a dictated
sentence can never pick up whatever the machine is playing.

Refs #18

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record & Transcribe now asks whether the meeting is in-person or online.
Online opens the loopback capture, adds a PC level meter beside the
microphone meter, and warns when no PC audio has arrived for 30 seconds —
the failure you would otherwise only discover after the meeting.

Settings gains a system audio device dropdown for when you switch between
headphones and speakers. Outside the desktop app the online option is
disabled: browser mode would need getDisplayMedia and a share prompt.

Refs #18

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renders a quiet 440 Hz tone on the default output and asserts the loopback
stream picks it up and reaches the mixed timeline. Ignored by default since
it needs audio hardware: cargo test --lib -- --ignored loopback

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@piyton
piyton requested a review from mojtabakarimi as a code owner August 28, 2026 11:27
@piyton
piyton merged commit 3efde28 into main Aug 28, 2026
1 check passed
@piyton
piyton deleted the feat/online-meeting-system-audio branch August 28, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opnemen en Transcriberen

1 participant