Skip to content

fix(blaze)!: reject incomplete reset and pool use - #2290

Merged
WeissonHan merged 1 commit into
alibaba:mainfrom
WeissonHan:fix/blaze/reject-incomplete-reset
Aug 16, 2026
Merged

fix(blaze)!: reject incomplete reset and pool use#2290
WeissonHan merged 1 commit into
alibaba:mainfrom
WeissonHan:fix/blaze/reject-incomplete-reset

Conversation

@WeissonHan

@WeissonHan WeissonHan commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Why

POST /v1/instances/{id}/reset previously returned success after changing
lifecycle metadata and placing the same running sandbox into a reusable pool.
It did not reset either the runtime or storage, so a later create request could
receive process or data state that the caller expected reset to discard.

Once incomplete reset is rejected, no supported path can safely produce a
reusable sandbox. Keeping the inactive pool implementation, settings, metrics,
and apparently successful management surface would continue to advertise a
capability that cannot meet its isolation contract.

What changed

  • Preserve the existing reset error boundary:
    • malformed identifiers return HTTP 400;
    • unknown instances return HTTP 404;
    • instances that are not running return HTTP 422;
    • running instances return HTTP 501.
  • Return the reset rejection before changing lifecycle state, persisted state,
    runtime ownership, or storage ownership.
  • Remove reusable-instance activation from sandbox creation. New instances
    always use the cold start path.
  • Keep these route shapes as compatibility endpoints that return HTTP 501:
    • GET /v1/pools
    • GET /v1/pools/{backend}/{class}
    • POST /v1/pools/{backend}/{class}/drain
    • PUT /v1/pools/{backend}/{class}/sizing
  • Remove the inactive pool implementation and stop publishing
    blaze_instances_resets_total, blaze_pool_hits_total, and
    blaze_pool_misses_total.
  • Reject unsupported pool configuration while preserving a narrow RPM upgrade
    path for files retained by %config(noreplace).

The only accepted daemon [pool] section is the exact historical package
default:

[pool]
default_warm_ttl = "30m"
gc_interval = "5m"

The only accepted policy [pool] sections must contain the exact six
historical fields and match one of these packaged policy identities:

Policy name Workload class min target max
agent-rl-default agent-rl 4 16 64
agent-tool-default agent-tool 2 8 32

Both also require enabled = true, warm_ttl = "30m", and
reset_mode = "full-recreate".

These accepted sections are ignored, omitted from serialization, and produce a
warning. They do not enable reusable instances. A missing or additional field,
a changed value or type, a different policy name or workload class, any other
daemon or policy [pool] section, and every occurrence of
storage.pool_size or storage.prefork remain validation errors.

At initial startup, a policy-loading failure obeys
policy.on_load_error: "fail" stops startup and "warn" starts with an
empty policy engine. Administrative and signal-driven reloads construct and
validate the replacement engine before publishing it; a failed reload leaves
the currently active policies unchanged.

Blaze continues to decode persisted Reset, Warm, and
start_path = "warm" values written by older releases. These values are
retained only for persisted-state cleanup: new runtime transitions cannot
enter Reset or Warm, and new instances always record a cold start. Startup
reconciliation treats non-terminal legacy records as cleanup candidates and
never reuses them. Failed cleanup leaves the affected record in
RecoveryRequired while reconciliation continues with other records.

Related issue

closes #2288

User impact

Clients requiring a fresh sandbox must stop relying on reset. Until complete
runtime-and-storage reset exists, they should successfully destroy the old
sandbox and create a new one.

RPM upgrades may continue to start when an administrator-modified
%config(noreplace) file still contains one of the exact packaged defaults
listed above. Operators should merge the corresponding .rpmnew files or
remove the legacy sections. Any actual pool tuning or other unsupported pool
setting is still rejected.

Monitoring rules and dashboards must stop querying the three removed metrics.
Direct blaze-core users must stop importing the removed pool interfaces.
Storage providers no longer implement pool draining, and callers can no longer
select a warm start for a newly constructed SandboxInstance.

Existing persisted state requires no conversion. Legacy reset and warm records
remain readable solely so startup reconciliation can release their owned
resources.

Risk and compatibility

  • Public CLI, API, configuration, or documented behavior changed
  • Privileged or security-sensitive behavior changed
  • Cross-component contract changed
  • Migration or rollback guidance is needed

This is an intentional breaking correction for reset clients, pool-management
clients, unsupported pool configuration, removed metrics, and direct users of
the removed Rust interfaces. The narrow package-default exception prevents an
unchanged historical RPM section from causing an otherwise compatible upgrade
failure; it does not preserve pool behavior.

Validation

Frozen source candidate:

  • Commit: adbe1c1220b426e3340877c0599f31d9c2f4237c
  • Declared parent: f7b0485b49f87fdaa7dc3f3251a58a80bd9f106a
  • Tree: 7d75cd5029ea9800e23d8b088c3caeaa1e798788

Current-main GitHub merge candidate:

  • Upstream main: ba20d94d19a73e5b781cab0c687d4034ad0d0dae
  • Merge commit: 3b5b4ec3aca098a7f0294f74a36282b51e3db1af
  • Parents: ba20d94d19a73e5b781cab0c687d4034ad0d0dae and adbe1c1220b426e3340877c0599f31d9c2f4237c
  • Merge tree: 20230d68b745dd80373a535048b155482e75a84e

Both exact trees were independently reconstructed from the public GitHub
source on native Linux x86_64 and matched the expected commit, parent, and
tree identities. Rust and Cargo 1.88.0 were used with locked dependencies,
offline Cargo execution, a fresh task-owned Cargo home, and an empty target
directory for every stage. Both trees passed:

  • cargo fmt --all -- --check;
  • default and all-feature locked metadata;
  • default and all-feature workspace builds for all targets;
  • default and all-feature strict Clippy for all targets with warnings denied;
  • default tests: 301 passed per tree;
  • all-feature tests: 316 passed per tree;
  • default and all-feature Rustdoc with warnings denied;
  • the unmodified documentation naming, English/Chinese parity, and
    relative-link checks; and
  • 17 focused reset, pool, configuration, legacy-state recovery, health, and
    metrics regressions per tree, each required to execute exactly one test.

The final run completed 1,234 full-suite test executions and 34 focused test
executions with no failures. The exact source and merge patches have the same
stable patch identity, and cleanup removed all task source, dependency cache,
target, and harness directories after archiving the logs.

All 26 hosted check runs for the published revision reached a terminal state
with no failures: Blaze, documentation, website, pull-request, and commit
message checks succeeded, while unrelated component and deployment jobs were
skipped by their change filters. The current GitHub check-run surfaces did
not expose a separate contributor-license status item.

Codex reviewed exact commit adbe1c1220 after publication and reported no
major issues. That review introduced no new inline thread or P1/P2 finding.

The final Linux evidence archive is
/tmp/pr2290-adbe-3b5b-exact-linux-gate-final-logs.tar.gz, with SHA-256
cb156a818b4080aa26abace6cd7d90f87292af2162fee7bf3308785dc68abb0b.

Documentation and rollback

The client, operator, and lifecycle compatibility contract is documented in
six existing paired documents:

  • src/blaze/README.md
  • src/blaze/README_zh.md
  • docs/user-guide/en/runtime/blaze.md
  • docs/user-guide/zh/runtime/blaze.md
  • src/blaze/docs/design/lifecycle-state-consistency.md
  • src/blaze/docs/design/lifecycle-state-consistency_zh.md

The lifecycle design pair now formally owns two related boundaries: complete
startup inventory publication, and serving-time rejection of reset or
reusable-instance operations that cannot preserve lifecycle, runtime, and
storage ownership. It defines the reset response matrix, the four reserved
pool-management responses, cold-only creation, compatibility-only decoding of
legacy Reset, Warm, and start_path = "warm" records, cleanup outcomes,
the retained health-response shape, and the removed metrics.

The four reader-facing documents give the exact accepted upgrade values. The
daemon section must contain only default_warm_ttl = "30m" and
gc_interval = "5m". The policy exceptions are limited to
agent-rl-default with workload class agent-rl and sizes 4/16/64, or
agent-tool-default with workload class agent-tool and sizes 2/8/32;
both require enabled = true, warm_ttl = "30m", and
reset_mode = "full-recreate". They also explain .rpmnew migration,
startup and reload failure behavior, and the recovery failure boundary: failed
legacy cleanup retains the in-memory record as RecoveryRequired and attempts
to persist that state; if persistence also fails, the warning reports the
additional error, the durable record may retain its previous state, and
reconciliation continues with other accepted records.

Migration requires:

  1. replacing reset-based reuse with successful destroy followed by create;
  2. merging .rpmnew files or removing legacy pool sections;
  3. removing unsupported pool settings and queries for removed metrics; and
  4. updating direct blaze-core users to stop using removed pool interfaces.

Reverting this commit requires no persisted-state conversion. It also restores
the false-success reset behavior and inactive pool surface, so updating clients
and configuration is safer than relying on rollback to recover those
operations.

Copy link
Copy Markdown
Collaborator Author

@codex review Please review exact head 7068096ec2792dc3f630726375f88dd7552f4d8f. This PR makes the incomplete sandbox reset endpoint return 501 before lifecycle, persistence, runtime ownership, pool membership, or metrics can change.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 7068096ec2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch from 7068096 to 4a544aa Compare August 10, 2026 08:25

Copy link
Copy Markdown
Collaborator Author

@codex review This PR rejects incomplete sandbox reset before lifecycle, ownership, pool, persistence, or metric mutation and keeps warm-pool setup test-only until complete reset exists. Please review the complete pull request: every commit, the cumulative diff against the base branch, and the implementation, tests, and documentation as one submitted change.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 4a544aa2ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan
WeissonHan marked this pull request as ready for review August 14, 2026 03:23
@WeissonHan
WeissonHan requested a review from casparant as a code owner August 14, 2026 03:23

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本次变更总体实现了 reset 未实现时的 501 返回和状态不变、更安全的行为。上面几点建议主要是关于 reset 状态约束语义和测试辅助函数命名清晰度,可酌情考虑。


🤖 Generated by QoderView workflow run

Comment thread src/blaze/crates/blazed/src/api.rs
Comment thread src/blaze/crates/blazed/src/api.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a544aa2ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/error.rs
@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch from 4a544aa to be904b1 Compare August 14, 2026 05:57

Copy link
Copy Markdown
Collaborator Author

@codex review This PR rejects incomplete sandbox resets before any state or ownership mutation while preserving existing identifier and lifecycle error semantics.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be904b1267

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/README.md Outdated
@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch 4 times, most recently from f139625 to 27debfa Compare August 14, 2026 06:21

Copy link
Copy Markdown
Collaborator Author

@codex review This PR makes the public reset endpoint reject incomplete resets without side effects, preserves existing identifier and lifecycle errors, and documents the exact protocol boundary.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 27debfa252

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch from 27debfa to 768b606 Compare August 14, 2026 06:28

Copy link
Copy Markdown
Collaborator Author

@codex review This PR makes the public reset endpoint reject incomplete resets without side effects, preserves existing identifier and lifecycle errors, documents the exact HTTP contract, and keeps internal error types crate-private.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 768b6066cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

WeissonHan commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up non-checkpoint API namespace work is tracked in #2577.

This pull request remains limited to correcting the false-success reset behavior on the existing route. #2577 will make the current non-checkpoint /v1/sandboxes routes canonical, keep the corresponding /v1/instances routes as explicit compatibility adapters, and add the canonical reset route only after #2290 has fixed its rejection contract.

The broader goal remains related to #2535, but checkpoint capture and its canonical route stay with #2472. No checkpoint route, namespace migration, compatibility-route removal, or deprecation date is included in #2290.

@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch from 768b606 to 90746b4 Compare August 14, 2026 09:35
@WeissonHan WeissonHan changed the title fix(blaze): reject incomplete sandbox resets fix(blaze)!: reject incomplete reset and pool use Aug 14, 2026
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review

@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch from 90746b4 to f6e19ae Compare August 14, 2026 10:12
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review

Copy link
Copy Markdown
Collaborator Author

@codex review This pull request rejects incomplete reset and inactive reusable-instance operations, preserves legacy state-file compatibility, and documents the public migration contract.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f6e19ae0ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/crates/blaze-core/src/config.rs Outdated
Comment thread src/blaze/crates/blaze-core/src/config.rs
Comment thread src/blaze/crates/blaze-core/src/lifecycle.rs

Copy link
Copy Markdown
Collaborator Author

@codex review Reject incomplete reset and inactive reusable-instance behavior while preserving exact packaged RPM defaults for upgrade compatibility.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f6f872396d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/blaze/README.md

WeissonHan commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

@casparant, the exact-head review found a documentation-placement conflict that now blocks #2290.

Section 5 of specs/documentation-standard.md requires protocol changes to add component design documentation. The current submission boundary excludes new or restored internal design documents and instead records the complete reset and pool compatibility contract in the paired Blaze READMEs and runtime user guides. I have left the review thread unresolved and have not claimed that this satisfies section 5.

Please confirm which boundary should govern. If section 5 must take precedence, the agreed submission plan must be revised before paired design documents can be added and the exact Linux and hosted gates repeated; I will not expand the pull request silently. If the four user-facing documents are sufficient for this corrective compatibility change, the current validated head can remain unchanged.

Reset changed only lifecycle metadata before returning the same runtime to an
instance pool. It did not reset runtime or storage, so callers could receive a
false success and later reuse state they expected to discard.

Return 501 for running reset requests while preserving 400, 404, and 422 for
malformed, missing, and invalid-state requests. Remove the inactive pool
implementation because reset was its only source of reusable instances.

Keep the four pool routes as 501 compatibility endpoints. Reject unsupported
pool settings, but accept and ignore only the exact defaults shipped in older
RPM configuration and policy files. This narrow exception prevents an
administrator-modified %config(noreplace) file from blocking the upgraded
binary before the operator can merge its .rpmnew replacement.

Retain decoding for legacy Reset and Warm states, including persisted
start_path = warm records, so startup cleanup can release their resources.
Remove the three metrics that no longer describe a supported operation.

This is a breaking correction for clients using the incomplete pool API,
unsupported pool configuration, or removed metrics. Complete reusable-instance
support remains out of scope and must arrive with sandbox creation that uses it.

Fixes: 1f0cfac ("feat(anvil): scaffold local orchestrator crate skeleton")
Signed-off-by: Weisson Han <wenshu.hx@linux.alibaba.com>
@WeissonHan
WeissonHan force-pushed the fix/blaze/reject-incomplete-reset branch from f6f8723 to adbe1c1 Compare August 15, 2026 11:48

Copy link
Copy Markdown
Collaborator Author

@codex review This PR rejects incomplete reset and reserved pool-management operations before lifecycle or resource ownership changes, retains only exact package-default upgrade compatibility, and documents the cold-only and legacy-recovery protocol in the paired Blaze lifecycle design.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: adbe1c1220

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@casparant The documentation-standard issue is now addressed in adbe1c1220 by extending the existing paired lifecycle design documents; no design document was added or restored. The exact source and current-main merge trees passed the full native Linux matrix, all hosted checks completed without failure, and Codex reported no major issues for the exact head. The pull request is ready for maintainer review.

Copy link
Copy Markdown
Collaborator Author

@casparant All eight review threads are now resolved. The exact head remains adbe1c1220b426e3340877c0599f31d9c2f4237c; its source and current-main merge candidate passed the complete Linux and hosted validation, and the exact-head Codex review reported no major issues. This pull request remains ready for maintainer review.

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: adbe1c1220

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@WeissonHan
WeissonHan merged commit 98d7952 into alibaba:main Aug 16, 2026
28 checks passed
@WeissonHan
WeissonHan deleted the fix/blaze/reject-incomplete-reset branch August 16, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[blaze] fix: reject incomplete sandbox resets

2 participants