Hotfix Soniox fallback keys for dev#3221
Conversation
📋 PR Review Helper📱 Mobile App Build⏳ Waiting for build... 🕶️ ASG Client Build⏳ Waiting for build... 🔀 Test Locallygh pr checkout 3221 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d18f4e8cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if ( | ||
| code === 401 || | ||
| lower.includes("invalid api key") || | ||
| lower.includes("invalid_api_key") || | ||
| lower.includes("bad api key") || | ||
| lower.includes("unauthorized") | ||
| ) { |
There was a problem hiding this comment.
If Soniox returns Soniox error 403: ... for an authorization/forbidden key, this classifier falls through to transient unless the text happens to include unauthorized; elsewhere the transcription manager already treats 401 and 403 as auth errors. That means a forbidden primary/fallback key is only cooled down for 10 seconds and then selected again for future streams instead of being disabled for the process as the new key-pool spec requires. Include code === 403 in the auth classification.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Leaving this open intentionally for follow-up. We are not broadening 403 into process-disable behavior in the ASAP hotfix.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| this.config, | ||
| throw ( | ||
| lastError ?? | ||
| new Error(`No available Soniox credentials: ${JSON.stringify(this.keyPool.describeAvailability())}`) |
There was a problem hiding this comment.
Last error blocks credential retries
Medium Severity
When stream creation tries every credential in the pool and none succeed, it always rethrows the last failure. If that message is a non-retryable Soniox auth error while other keys are only on short cooldowns, TranscriptionManager stops retrying instead of waiting for a recoverable credential.
Reviewed by Cursor Bugbot for commit b2ee0a2. Configure here.
There was a problem hiding this comment.
Leaving this open intentionally for follow-up. The last-error masking case is valid but outside the narrowed prod outage fix; the pushed fix covers the spend-cap quota path.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8ab0ba7. Configure here.
| lower.includes("bad api key") || | ||
| lower.includes("unauthorized") | ||
| ) { | ||
| return { kind: "auth", cooldownMs: Number.POSITIVE_INFINITY, disabled: true }; |
There was a problem hiding this comment.
HTTP 403 not auth disabled
Medium Severity
classifySonioxCredentialFailure treats Soniox 401 and several auth phrases as auth and disables the credential, but 403 and typical “forbidden” wording are omitted. Those failures fall through to transient with a short cooldown, so forbidden keys stay in rotation instead of being disabled for the process as the spec requires.
Reviewed by Cursor Bugbot for commit 8ab0ba7. Configure here.
There was a problem hiding this comment.
Leaving this open intentionally for follow-up. We are not broadening 403 into process-disable behavior in the ASAP hotfix.
|
Related Soniox fallback PRs:
Current prod/main hotfix head: e06781b. |
1 participant
Summary
Backports the Soniox fallback key hotfix to
devfor legacy Cloud v1.Why
The primary Soniox org/key can hit account-level limits, causing every new transcription or translation stream to fail through the same exhausted credential. This branch mirrors the
mainhotfix so dev has the same operational behavior.What changed
SONIOX_FALLBACK_API_KEYSsupport for transcription and translation.SONIOX_API_KEYas primary while healthy.cloud/issues/108-soniox-fallback-api-keys/.Validation
bun test cloud/packages/cloud/src/services/session/soniox/__tests__/SonioxKeyPool.test.tsgit diff --check HEAD~1..HEADcd cloud && bun run buildNote:
cd cloud/packages/cloud && bun run buildcurrently fails ondevwith local workspace resolution for@mentra/display-utils, unrelated to this Soniox hotfix diff.Note
Medium Risk
Changes live Soniox stream creation and retry paths for all sessions; misconfigured keys or cooldown logic could affect STT/translation availability, though behavior is scoped to Soniox and covered by new tests.
Overview
Adds
SONIOX_FALLBACK_API_KEYS(comma-separated) so Cloud v1 can rotate across multiple Soniox org keys when the primary hits limits, instead of retrying the same exhausted credential.Introduces a shared
SonioxKeyPool: preferSONIOX_API_KEY, round-robin fallbacks, dedupe, per-key cooldowns from classified failures (concurrency, rate limit, quota, auth disable, transient), and SHA-256 fingerprints in logs only. Soniox transcription and translation providers loop credentials on stream creation (SDK + WebSocket), cache one SDK client per credential, and record runtime stream failures against the active key.TranscriptionManagerandTranslationManagertreat “no available credentials” and capacity-class Soniox errors as retryable so new streams can pick another key. Default Soniox realtime model is bumped tostt-rt-v5. Includes issue docs andSonioxKeyPoolunit tests.Reviewed by Cursor Bugbot for commit a157880. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Backports the Soniox fallback key hotfix to
devfor Cloud v1. Adds a shared credential pool with retry/rotation, tightens quota cooldown handling, and defaults Soniox tostt-rt-v5to reduce failed stream creations.New Features
SONIOX_FALLBACK_API_KEYS(comma‑separated) with parsing and dedupe.SonioxKeyPool: prefers primary, round-robins fallbacks, classifies failures (auth, concurrency, rate_limit, quota, transient) with per-key cooldowns; quota uses a long cooldown, and success no longer clears an active cooldown; logs only SHA‑256 key fingerprints.TranscriptionManagerandTranslationManagerfor Soniox capacity/quota errors and “No available Soniox (translation) credentials”.stt-rt-v5across transcription and translation; added tests and docs undercloud/issues/108-soniox-fallback-api-keys/.Migration
SONIOX_FALLBACK_API_KEYSin.envwhere multiple Soniox keys are available. No other changes required.Written for commit a157880. Summary will update on new commits.