Skip to content

Conversation

@toubatbrian
Copy link
Contributor

@toubatbrian toubatbrian commented Oct 24, 2025

Problem

The Silero VAD plugin was intermittently throwing unhandled TypeError: Invalid state: Writer is not bound to a WritableStream errors when plugin close.

Root Cause

A race condition exists between the VAD inference loop and the stream shutdown process:

  1. The VAD task runs in a continuous loop, checking while (!this.closed) before processing frames
  2. When a participant disconnects, the base class close() method:
    • Calls outputWriter.releaseLock() to unbind the writer from the stream
    • Sets closed = true
  3. These operations are not atomic, creating a window where the inference loop can pass the !this.closed check but then attempt to write to an already-released writer

Race Condition Timeline:

VAD Task                     |  Close Task
-----------------------------|---------------------------
Check: !this.closed → true   |
                             |  outputWriter.releaseLock()
                             |  closed = true
outputWriter.write(...) ❌   |
→ Writer is not bound!       |

Solution

Implemented graceful error handling following the established pattern in deferred_stream.ts.

@changeset-bot
Copy link

changeset-bot bot commented Oct 24, 2025

🦋 Changeset detected

Latest commit: 4db5c8f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@livekit/agents-plugin-silero Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Contributor

@Shubhrakanti Shubhrakanti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mention that the Close Thread calls outputWriter.releaseLock() .... closed = true but we don't multi thread? It's asynchronous. Are you sure this is the root cause?

Are you able to reliably reproduce this issue? If so can you list the steps?

@toubatbrian
Copy link
Contributor Author

You mention that the Close Thread calls outputWriter.releaseLock() .... closed = true but we don't multi thread? It's asynchronous. Are you sure this is the root cause?

Yes, I'm pretty sure. This is "async interleaving", there are await inside the this.#task promise:

await this.inputReader.read() 
await this.#model.run(inferenceData) 

both yield control back to the event loop.

Here's how the race condition occurs:

  1. VAD inference loop passes the while (!this.closed) check
  2. Execution reaches one of the await points → yields control to event loop
  3. While yielded, event loop processes a participant disconnect event
  4. This triggers close() which runs synchronously:
  5. this.outputWriter.releaseLock() unbinds writer
  6. Event loop resumes the VAD task after the await completes
  7. Execution continues past the !this.closed check but before checking again, code attempts this.outputWriter.write()Error: Writer is not bound

Comment on lines +28 to +43
function isWriterReleaseError(e: unknown): boolean {
if (e instanceof TypeError) {
// Check for ERR_INVALID_STATE error code (most reliable)
if ('code' in e && e.code === 'ERR_INVALID_STATE') {
return true;
}

// Fallback to message checking for compatibility
const message = e.message;
return (
message.includes('Writer is not bound to a WritableStream') ||
message.includes('WritableStream is closed')
);
}
return false;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this hacky? Should we have another mechanism of synchronization?

Copy link
Contributor Author

@toubatbrian toubatbrian Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have similar approach in defered_stream.ts

export function isStreamReaderReleaseError(e: unknown) {
const allowedMessages = [
'Invalid state: Releasing reader',
'Invalid state: The reader is not attached to a stream',
];
if (e instanceof TypeError) {
return allowedMessages.some((message) => e.message.includes(message));
}
return false;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error we're catching (ERR_INVALID_STATE: Writer is not bound) is essentially Node.js telling us the stream is shutting down. It's not really an exceptional condition, it's an expected part of the shutdown sequence that we need to handle gracefully. The safeWriteEvent() method makes this explicit: it catches expected shutdown errors and signals the loop to exit cleanly, while still throwing any truly unexpected errors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this happen in llm/stt/tts as well?

@toubatbrian toubatbrian requested a review from davidzhao October 24, 2025 18:28
Copy link
Contributor

@Shubhrakanti Shubhrakanti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@toubatbrian I think this is what you mean. We have a to await the model to actually run during which the outputWriter might be closed.

(model run)

const p = await this.#model
.run(inferenceData)
.then((data) => this.#expFilter.apply(1, data));

(output writer definition)

protected outputWriter: WritableStreamDefaultWriter<VADEvent>;

Is there a simpler solution where the base vad class has a method that wraps this.outputWriter.write with a if (!this.closed)? Then every plugin just calls this method and doesn't have to worry about the internals of outputWriter?

@toubatbrian
Copy link
Contributor Author

Is there a simpler solution where the base vad class has a method that wraps this.outputWriter.write with a if (!this.closed)? Then every plugin just calls this method and doesn't have to worry about the internals of outputWriter?

@Shubhrakanti Great question! Let's definitely explore that idea later. For now let's focus on fixing the VAD issue within the scope of this PR to unblock our customer first. We can revisit and discuss a cleaner solution afterward.

@Shubhrakanti
Copy link
Contributor

Shubhrakanti commented Oct 24, 2025

Is there a simpler solution where the base vad class has a method that wraps this.outputWriter.write with a if (!this.closed)? Then every plugin just calls this method and doesn't have to worry about the internals of outputWriter?

@Shubhrakanti Great question! Let's definitely explore that idea later. For now let's focus on fixing the VAD issue within the scope of this PR to unblock our customer first. We can revisit and discuss a cleaner solution afterward.

All we need to do is add one method to the base vad class

writeOutput(event: VADEvent) {
    if (!this.closed) {
       this.outputWriter.write(event)
    } else {
     // you can either throw an error or have a boolean which returns if the write was successful or not. Or just do nothing and the next iteration of `this.#task` in silerio.vad the main loop will break since `this.close == true`
   ....
}

and then you replace this.outputWriter.write( -> this.writeOutput in sliero/vad.ts.

It's fine if this goes out Mondaynext week. I'd rather have it done correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants