Repository navigation
alerting: retry transient Kimi requests and raise Full CI read timeout - #120
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The Full CI analysis for build #86943 (2026-09-03 02:00 UTC fire) failed 24 minutes in with "kimi API request failed: The read operation timed out". A single completion exceeded the 300s urllib read timeout, the transport raised, and the whole analysis was discarded with no retry, so no report reached Slack. Classify transport failures: read timeouts, connection errors, and HTTP 429/5xx raise TransientTransportError; other 4xx and malformed bodies stay permanent. KimiCodeRunner retries transient failures up to three times with 5s then 15s backoff, never sleeping past the whole-analysis deadline, and records the attempt count in the error when it gives up. Add a per-request timeout knob. The Full CI analyzer now defaults to 900s per completion via KIMI_REQUEST_TIMEOUT_SECONDS; Main CI keeps 300s and gains the retry. Document both knobs in alerting/docs/full-ci.md. Signed-off-by: Thang Nguyen <thang.nguyen@inferact.ai> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Thangnguyenvn98
force-pushed
the
agent/kimi-transient-retry
branch
from
September 3, 2026 03:42
e819e2c to
99e6e86
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Full CI analysis for build #86943 (2026-09-03 02:00 UTC timer fire) failed 24 minutes in:
One completion exceeded the 300s
urllibread timeout,UrllibTransportraised, and the analyzer discarded the entire run. No report row, no outbox entry, no Slack message. The build stayed pending until a manualsystemctl start alerting-full-ci.service.What
TransientTransportErrorfor read timeouts, connection errors, HTTP 429 and 5xx; other 4xx and malformed JSON remain permanentAnalyzerError.KimiCodeRunner._requestretries transient failures up to 3 times with 5s/15s backoff and refuses to sleep past the whole-analysis deadline. The final error carries the attempt count solast_errorin Postgres is self-explanatory.KIMI_REQUEST_TIMEOUT_SECONDS, default 900 for the Full CI analyzer (worker.py,build_full_ci_analysis_runtime). Main CI keeps 300s per request and gains the retry.alerting/docs/full-ci.mdlists both knobs and the retry policy.Verification
pytest: 194 passed (5 new intests/test_kimi.py— retry-then-succeed with[5.0, 15.0]sleeps, give up after 3 attempts, permanent 400 not retried, retry refused near the deadline, urllib classification).ruff check .: clean.mypy --strict: clean on every touched file. Remaining errors (control.py:92,analyzer.py:890,tests/test_worker.py:247-248) are pre-existing onmain.Deploy
Needs the worker's
/opt/alerting/sourcerefreshed viadeploy/aws/install.shafter merge (operations doc §9).🤖 Generated with Claude Code