Release 0.1a prep: Ascend Technical Preview landing page - #140
Open
Ray-RP wants to merge 50 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NPU + TP=1 no longer bootstraps torch.distributed. `_init_communication` now short-circuits to return None on that path, and `_sync_get_memory` and `shutdown` guard their all_reduce / destroy_process_group calls on the resulting nullable group. CUDA paths and the NPU TP>1 HCCL path are unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The outer @nvtx_annotate("Sampler") decorator already brackets this method
via the cross-device-safe shim in minisgl.utils.torch_utils. The redundant
inline `with torch.cuda.nvtx.range("Sampler"):` raised
`RuntimeError: NVTX functions not installed.` on non-CUDA Torch builds
(e.g. the Ascend container), blocking prefill on NPU.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scheduler.__init__ called torch.cuda.Stream(device=...), torch.cuda.stream,
and torch.cuda.set_stream unconditionally, and shutdown()/run_forever called
torch.cuda.synchronize / torch.cuda.current_stream — all of which raise
ValueError("Expected a cuda device") when device_type is "npu". Route every
stream primitive through the shared minisgl.utils.device_runtime dispatch layer
that Engine already uses, so cuda/npu/cpu hosts all follow the same call graph
via engine.device_type.
Adds a new stream_context(device_type, stream) helper — the only stream
primitive that was still missing from device_runtime — dispatching to
torch.cuda.stream, torch.npu.stream (with lazy torch_npu import), or
contextlib.nullcontext respectively.
Tests:
* test_device_runtime.py: fake torch fixture gains .stream context factory,
four new stream_context cases (cuda / npu / cpu / unknown), public-surface
pin extended with stream_context.
* test_scheduler_device_backend.py (new): structural check that scheduler.py
contains no raw torch.cuda.{Stream,stream,set_stream,current_stream,
synchronize} reference and imports the five device_runtime helpers;
behavioural check that Scheduler.__init__ never touches torch.cuda.Stream
on the NPU branch (raw APIs booby-trapped); behavioural check that
Scheduler.shutdown routes through synchronize_device with the engine's
device_type; behavioural check that the CUDA dispatch path is preserved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gate 1.11h. SchedulerIOMixin.sync_all_ranks() unconditionally called
self.tp_cpu_group.barrier().wait(), which crashed with AttributeError on
any single-rank host (TP=1 NPU, offline runs, ...) because
Engine._init_communication returns None when tp_info.size == 1 and no
distributed backend was ever brought up. That crash surfaced during
Scheduler.shutdown() and blocked the Gate 1.11f two-token probe from
reaching a clean exit.
Minimal fix: early return when self.tp_cpu_group is None; preserve the
original barrier().wait() behaviour whenever a real ProcessGroup is
attached. The guard is orthogonal to device type (per Gate 1.11h scope)
so it holds for CUDA, NPU, and CPU alike.
New tests in tests/misc/test_scheduler_sync_all_ranks.py pin:
* sync_all_ranks with tp_cpu_group=None never touches a barrier;
* with a real-looking group, barrier and wait are each called exactly
once and in order (TP>1 path unchanged);
* Scheduler.shutdown on a TP=1 NPU still routes device synchronize
through minisgl.utils.device_runtime and still tears down the engine;
* with a real group, shutdown order remains
synchronize_device -> barrier -> wait -> engine.shutdown.
Local regression (container 998ce5ba6e5e):
test_scheduler_sync_all_ranks.py 4 passed
test_scheduler_device_backend.py 5 passed
Combined 9 passed
test_device_runtime.py standalone 59 passed
(Cross-file `torch_npu in sys.modules` collision when device_runtime
tests run after scheduler tests is pre-existing and out of Gate 1.11h
scope — device_runtime tests remain untouched.)
910B1 rerun:
Gate 1.11g Scheduler init smoke PROBE_RESULT=PASS
- scheduler.stream/engine.stream both torch_npu.npu.streams.Stream
- dist_initialized = False
- shutdown_ok = True
Gate 1.11f scheduler-driven 2-token probe PROBE_RESULT=PASS
- reply_log tokens = [15087, 11], reply[1].finished = True
- batches: prefill positions=[0,1,2,3] out_loc=[0,1,2,3]
decode positions=[4] out_loc=[4]
- KV writes: slots 0..4 → 28 layers on both K and V; slots 5..15 = 0
- counts: model_forward=2, sampler_sample=2, prepare_metadata=2
store_kv=56 (28 layers x 2 batches), fia=56
init_process_group=0, all_reduce=0, destroy_process_group=0
- dist_initialized (after run) = False
- shutdown_ok = True
Push pending — remote git network unreachable from this workstation
(same condition observed for the Gate 1.11g commit 40bc57d).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gate 1.12a. Records the frozen evidence for single-request eager inference on Ascend 910B1 at HEAD 0206c54: * Hardware / software matrix (Ascend 910B1, CANN 8.5.1, torch 2.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, Qwen3-0.6B fp16). * Explicit in-scope / out-of-scope surface. Gate 1 covers TP=1, eager forward, greedy sampling, single request, npu_fia attention. * Chronological list of the Ascend-port commits from 806e1b7 through 0206c54 (portable device/distributed runtime, KV layout, FIA, layer dispatch, Gate 1.11 fixes). * Frozen prefill/decode token invariants: input [3,7,11,15] deterministic two-token greedy output [15087, 11] with reply[1].finished=True. Positions/out_loc/cached_len/device_len contract per batch. * KV / FIA contract: 28 layers x 5 slots = 140 K writes and 140 V writes; slots 0..4 fully written, slots 5..15 untouched; FIA and store_kv each called 56 times; prepare_metadata 2 times. * Distributed contract: init_process_group / all_reduce / destroy_process_group all called 0 times for TP=1 NPU; shutdown returns cleanly. * Regression evidence: 5 test modules, 79/79 passed standalone at HEAD 0206c54 (test_device_runtime 59, test_engine_tp1_nodist 7, test_sampler_no_cuda_nvtx 4, test_scheduler_device_backend 5, test_scheduler_sync_all_ranks 4). Documented harness cross-file ordering limitations without altering runtime. * Known gaps for later gates: TP>1 forward, continuous batching, graph capture, non-greedy samplers, radix reuse, cross-file test isolation, ZMQ tokenizer path. * Reproduction outline pointing at tag gate1-single-request-eager. No runtime code touched by this commit. No temporary probes, private paths, or workstation credentials committed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds SamplingParams.stop_token_ids (immutable tuple default) and wires a
membership check into Scheduler._process_last_data after req.append_host,
so the stop token itself is retained in the output sequence. The check
runs independently of ignore_eos — ignore_eos only silences the
tokenizer's EOS, it does not disable user-declared stop tokens. An empty
tuple preserves pre-2.1c behaviour verbatim.
Verified on 910B1 with input_ids=[3,7,11,15], stop_token_ids=(11,):
- tokens=[15087, 11], finished=[False, True]
- remain_len=6 (>0) proves termination is stop-token, not max_tokens
- token 11 != eos_token_id (151645) proves it is not EOS
- 2 batches only (prefill + decode), no third batch produced
- KV page 0 slots 0-4 written on all 28 layers; slot 5 empty
- allocator fully restored; torch.distributed never touched
Test coverage (17 new tests, all pass):
- test_sampling_params_stop_tokens.py: field contract, tuple default,
identity sharing, orthogonality with is_greedy, kwarg order
- test_scheduler_stop_tokens.py: AST invariants (attribute reference,
`in` compare, not nested under ignore_eos) plus behaviour rows for
default empty tuple, stop-token hit + retention, ignore_eos still
honours stop_token_ids, non-match keeps running, max_tokens and EOS
paths unchanged.
RadixPrefixCache._tree_walk -> fast_compare_key runs on every prefix match on every device, so apache-tvm-ffi is a cross-device runtime requirement. Leaving it in [cuda] made clean Ascend installs fail with ModuleNotFoundError: No module named 'tvm_ffi' at the first prefill scheduling call (surfaced in Gate 2.1e). Move the pin (>=0.1.4) into [project].dependencies and drop it from the cuda extra; add structural tests locking the cross-device placement and pin.
Records the PASS verdict for Gate 2.1 (single-request multi-step scheduler on Ascend 910B1): supported scope, cross-page KV contract, stop-token semantics, Radix reuse+eviction evidence, the apache-tvm-ffi base-dep requirement, and the identified follow-up gap (multi-request batching). Frozen at commit 846c16f.
Gate 2.2b confirmed the underlying FIA operator accepts B>=1 equal-length prefill/decode directly; only the wrapper's single-request guard needed lifting. - prepare_metadata now accepts any B>=1 as long as every real request in batch.padded_reqs shares extend_len, cached_len and device_len. Any variance (ragged batch) raises NotImplementedError with a message identifying which request diverged — no silent fallthrough into a mis-shaped metadata build. - FIAMetadata carries an explicit batch_size field; actual_seq_lengths and actual_seq_lengths_kv are Python lists of length B with the shared value. block_table is now [B, num_blocks] built row-by-row via torch.stack under the same stride-then-divide algorithm. - forward reshapes the flat query [B*S, Hq, D] to BSND [B, S, Hq, D], builds a shared [S, padded_kv_len] causal mask offset by the common cached prefix (or None on the decode path), then reshapes FIA's output back to the caller's flat layout. store_kv still receives the concatenated batch.out_loc so per-request page ranges stay disjoint. - FIAMetadata.get_last_indices now returns the flat-buffer offset of each request's last token — mirrors cu_seqlens_q[1:1+bs]-1 semantics. Tests: 53 hermetic cases covering B=1 continuity, B=2 equal-length prefill/decode metadata, block_table row independence, ragged extend_len/cached_len rejection, forward-side shape-mismatch guard, shared causal-mask visibility, disjoint store_kv slot ranges, unchanged CUDA/FA/FI/TRTLLM registration and metadata sources. Rerun of the Gate 2.2a spec on 910B1 with A=[3,7,11,15] → [15087,11] and B=[15,15,3,15] → [15,15]: 1 prefill batch of size 2 + 1 decode batch of size 2; model_forward=2, FIA=56, store_kv=56; A → physical page 0, B → physical page 1 (disjoint); allocator baseline restored on shutdown; torch.distributed uninitialised. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… 2.2f) Extend AscendFIABackend to accept two additional batch shapes on top of the equal-length B>=1 path from Gate 2.2c: 1. Ragged prefill (all cached_len == 0, extend_len may vary): pack the flat query into padded BSND [B, max_query_len, Hq, D], build a per-batch causal mask [B, 1, max_query_len, padded_kv_len] with padded query rows fully masked, and unpad the output back to the flat [sum_q, Hq, D] shape. Block table rows are right-padded with 0 to a shared max_blocks width. 2. Pure-decode ragged (all extend_len == 1, cached_len may vary): reshape flat [B, Hq, D] to [B, 1, Hq, D], pass atten_mask=None, and let per-request KV lengths flow through actual_seq_lengths_kv. FIAMetadata gains query_seq_lens / kv_seq_lens / max_query_len / query_offsets. query_seq_len / kv_seq_len stay populated only under the equal-length path (None under ragged / pure-decode mixed). Ragged prefill with any non-zero cached_len and extend_len > 1 remains NotImplementedError. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Records the multi-request batching contract locked at 3d6b3ed: equal-length prefill, ragged prefill with no cached prefix, pure-decode with mixed cached_len, dynamic admission, and batch grow/shrink over three requests. Notes the currently rejected ragged-with-cached-prefix shape and the next gate's follow-ups (cancellation, IPC, preemption, stress).
…Gate 2.3b) CacheManager.allocate_paged now returns an AllocationToken that snapshots exactly what it mutated: the newly-drawn physical pages and the pre-write contents of every page_table region it is about to overwrite. rollback_allocation undoes both without touching prefix pages, cache handles, refcounts, or the running/pending sets. Scheduler._prepare_batch wraps every step after allocate_paged in try/except. On any raise (attn metadata refusal, sampler prep failure, etc.) the token rolls back allocator state, the batch scratch attrs (positions, out_loc, padded_reqs, attn_metadata) are scrubbed, and the original exception propagates unchanged. Out of scope, intentionally left for later gates: - prefix cache lock/unlock (owned by PrefillAdder before _prepare_batch) - table_manager.allocate/free (same reason) - model.forward / sampler.sample raises after _prepare_batch returns Tests: 5 hermetic CPU tests in tests/misc/test_scheduler_prepare_batch_txn.py exercise real CacheManager + real allocate_paged with a NaivePrefixCache, covering no-prefix rollback, with-prefix rollback, multi-req different page counts, success regression, and rollback-then-success. 910B1 probe: cached-prefix ragged prefill triggers the FIA backend's NotImplementedError; free_slots multiset, page_table contents, and available_size are all bit-equal to their pre-allocate_paged snapshot; a subsequent single-request prefill+decode succeeds. Regression: tests/misc/test_ascend_fia_backend.py 69 passed; Gate 2.2i equal-length smoke and Gate 2.2f ragged smoke both PASS. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te 2.3c)
Engine.forward_batch previously advanced req.cached_len / req.device_len
via req.complete_one() BEFORE sampler.sample() ran. A raise inside the
sampler therefore left request state one step ahead of the token that
never made it into token_pool — subsequent scheduling rounds would then
allocate/overwrite the wrong page_table slot for that request.
New order inside forward_batch:
1. model.forward() / graph_runner.replay() -> logits
2. sampler.sample(logits[:batch.size], args) -> next_tokens_gpu (int32)
3. Basic shape validation: next_tokens_gpu must be 1-D with batch.size
entries. Anything else raises RuntimeError before we touch req state.
4. All-or-nothing commit: for req in batch.reqs: req.complete_one().
complete_one is pure Python attribute arithmetic on the Req
dataclass and cannot itself raise, so this loop either advances
every real request or none of them.
5. Copy to CPU, record event, return ForwardOutput.
padded_reqs are not touched by this loop — only real batch.reqs commit,
matching Gate 2.2 semantics.
Explicitly out of scope for Gate 2.3c: model.forward failure paths (KV
already written), page release, Scheduler-level exception envelope,
abort ack, overlap-loop race, shutdown drain.
Tests: 5 hermetic tests in tests/misc/test_engine_forward_sampler_atomic.py
using a real Engine shell (__new__ + swapped-in ctx/model/sampler) and
real Req dataclasses. Covers B=1 sampler failure, B=2 sampler failure
without partial commit, B=2 sampler success (baseline output preserved),
sampler wrong-shape output, and padded-batch commit isolation.
910B1 fault-injection probe: real Qwen3 model.forward + 28 layers of FIA
run to completion, monkeypatched Sampler.sample raises
RuntimeError("gate23c injected sampler failure"). Post-raise state
identical to pre-forward: device_len 8->8, cached_len 0->0, extend_len
8->8, table_idx 7->7, page_table row bit-equal, available_size
271856->271856, free_pages 16991->16991. Original exception type +
message propagates unchanged.
Regression: hermetic scheduler-txn (5) + FIA backend (69) all pass;
fresh-process single-request and B=2 ragged prefill+decode produce
tokens identical to Gate 2.2 baseline ([82, 13] etc.).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scheduler.shutdown() previously walked through synchronize_device +
sync_all_ranks + engine.shutdown() without touching queues. Any request
still in pending_list or decode_manager.running_reqs left its table_idx
and its KV pages "owned" — cache_manager.check_integrity() would fail
if run afterwards, and the process exited with occupied slots.
New shutdown sequence:
1. synchronize_device(device_type) — wait for in-flight work
2. sync_all_ranks() — TP rendezvous
3. _drain_requests() — reclaim queue state
4. engine.shutdown() — tear down comms/graph
_drain_requests is the new internal helper. Contract:
* Pending waiting reqs (no ChunkedReq): removed from pending_list,
no table/KV touched (there is nothing to free).
* Pending reqs carrying a ChunkedReq: the ChunkedReq goes through
_free_req_resources exactly once (table_idx + KV via
cache_manager.cache_req(finished=True)). For requests whose
device_len exceeds cached_len, the tail region
page_table[table_idx, page_align(cached_len):page_align(device_len)]
is _free()'d BEFORE the standard release path, so pages that
allocate_paged wrote but complete_one never committed are also
reclaimed. Page math mirrors allocate_paged (div_ceil rounding).
* Running decode reqs: iterated in uid-sorted order (Set iteration
is nondeterministic) so review + reproducibility hold; each goes
through _free_req_resources exactly once.
* finished_reqs guard set is cleared so a subsequent drain does not
see a phantom "already freed" mask.
Dedup + idempotency:
* Freed Reqs are tracked by id() within a single drain call — a Req
accidentally referenced from both pending_list (as ChunkedReq) AND
running_reqs is freed once, not twice.
* A second call to _drain_requests is a no-op: both containers are
already empty; freed_ids is a fresh empty set that never fires.
* No abort ack, no final DetokenizeMsg — this Gate only guarantees
backend resource cleanup.
Tests: tests/misc/test_scheduler_shutdown_drain.py (8 hermetic tests):
A waiting-only pending -> allocator untouched
B pending with ChunkedReq -> table + KV reclaimed
C running-only decode reqs -> table + KV reclaimed
D mixed pending + chunked + running -> everything back to baseline
E shared Req in two containers -> freed once, no double-free
F drain twice -> second call is a pure no-op
G empty scheduler drain -> allocator untouched
H shutdown() invokes the sequence sync_device -> sync_ranks ->
drain -> engine_shutdown in that exact order
910B1 shutdown probe:
Pre-shutdown: uid=A running (table_idx=7, cached_len=9, device_len=10),
uid=B pending, table_free=[0..6], available_size=271840, free_pages=16990.
Post-shutdown: running=[], pending=[], table_free=[0..7] (baseline),
available_size=271856 (baseline), free_pages=16991 (baseline),
distributed_initialized=False, no exception.
Fresh-process regression: single-request tokens [82, 13] and B=2 ragged
tokens [4999, 13] / [16, 15] — identical to the Gate 2.3c baseline.
Regression suites: scheduler-txn (5) + engine-sampler-atomic (5) +
shutdown-drain (8) + FIA (69) = 87 passed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce an explicit fence in Scheduler.overlap_loop so an AbortBackendMsg
that arrives for a uid whose forward has completed but whose CPU-side
_process_last_data has NOT yet run cannot race the batch's post-processing.
Two new Scheduler state sets:
* inflight_uids — populated at overlap_loop entry from
last_data.batch.reqs, cleared at loop exit.
* deferred_abort_uids — tracks aborts held back for the current fence
window.
_process_one_msg's AbortBackendMsg branch now checks msg.uid against
inflight_uids: if present, the req is yanked out of both decode_manager
and prefill_manager (so it cannot enter the next scheduled batch),
added to deferred_abort_uids, and its resources are LEFT alone.
Otherwise the existing immediate-free path runs unchanged.
_process_last_data skips its per-req token-emit / finish path for any
uid in deferred_abort_uids — no stale DetokenizeMsg, no append_host.
A new _apply_deferred_aborts helper runs from overlap_loop AFTER
_process_last_data completes. It walks last_data.batch.reqs, id()-dedups
(same pattern as _drain_requests), and calls _free_req_resources exactly
once per deferred uid. Then clears deferred_abort_uids; overlap_loop
clears inflight_uids immediately after.
normal_loop never populates inflight_uids so its abort path is untouched.
Coverage:
* tests/misc/test_scheduler_overlap_abort_fence.py — 7 hermetic tests
covering non-inflight (immediate) abort, inflight defer + apply,
unschedulability of deferred uid, stale-reply suppression, duplicate
abort idempotency, B=2 partial abort keeps survivor, and normal_loop
invariant. All pass with the real CacheManager / TableManager /
PrefillManager / DecodeManager under Scheduler.__new__.
* 910B1 overlap probe (A+B, max_tokens=4, abort A mid-fence) confirms
fence branch fired (abort_uid_in_inflight=True, deferred={A}), A
absent from all post-abort batches, A's tick-2 decode reply
suppressed while its pre-abort prefill reply is preserved, B's
token stream matches the single-request baseline
[16, 15, 15, 15], and A is freed exactly once. Post-shutdown
allocator state matches pre-run baseline.
* Regression: Gate 2.3b + 2.3c + 2.3d hermetic suites (25 tests) all
pass, tests/misc/test_ascend_fia_backend.py (69 tests) passes on
NPU, and the fresh-process baseline probe reproduces the baseline
single-request and B=2 ragged prefill+decode outputs unchanged.
Out of scope (per Gate 2.3e brief): abort ack protocol,
model.forward/FIA exception recovery, sampler failure continuation,
shutdown drain semantics, normal_loop behavior changes, ExitMsg
protocol changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce a dedicated abort-ack path so a Frontend abort resolves via a
scheduler→tokenizer→frontend acknowledgement instead of a synthesised
DetokenizeMsg with next_token=-1.
Message types
* message/tokenizer.py: AbortAckMsg(uid) — Scheduler → Tokenizer,
reuses the BaseTokenizerMsg channel already carrying DetokenizeMsg.
* message/frontend.py: AbortAckReply(uid) — Tokenizer → Frontend,
parallel to UserReply so listen() dispatches by isinstance.
Scheduler emission points (all four cases end in exactly one ack)
* waiting/pending abort — one ack after pending removal, no free.
* running non-inflight abort — one ack AFTER _free_req_resources.
* overlap deferred abort — one ack AFTER _apply_deferred_aborts
performs the single id()-deduped free.
* unknown uid — one idempotent ack, no free.
Acks accumulate in Scheduler._pending_abort_acks and are flushed at
the tail of overlap_loop / normal_loop via _flush_pending_acks so the
tick's UserReplies go out before its AbortAckMsgs. Non-primary ranks
route through _reply_tokenizer_rank1 (no-op), no new rank guard needed.
Tokenizer worker
* server.py partitions the inbound BaseTokenizerMsg batch into a
fourth bucket (abort_ack_msg), forwards them 1:1 as AbortAckReply
on send_frontend. Assert count matches include the new bucket.
Frontend state machine
* FrontendManager.abort_user(uid): drops the sleep + premature dict
delete; marks uid in abort_pending, allocates abort_pending_events,
sends AbortMsg, returns. Callers awaiting synchronous proof use the
new wait_for_abort_ack coroutine. Client-disconnect path does NOT
block on the ack — the request task is free to finalise.
* FrontendManager.listen: AbortAckReply → cleanup ack_map + event_map,
remove from abort_pending, fire abort_pending_events. UserReply for
a uid in abort_pending is dropped (late-token suppression). Unknown
uid on either message type is a silent no-op (idempotency).
* wait_for_ack: tolerates concurrent cleanup by the ack handler and
exits cleanly when ack_map[uid] disappears mid-await.
* _unwrap_msg: return type widened to List[BaseFrontendMsg] so
AbortAckReply flows through the same channel.
I/O plumbing
* scheduler/io.py: reply signatures widened to List[BaseTokenizerMsg].
DetokenizeMsg import dropped (not referenced after the widening).
Regression + tests
* tests/misc/test_scheduler_abort_ack.py (8 hermetic tests A-H):
A waiting abort → no free, one ack
B running non-inflight → free precedes ack
C overlap deferred → ack only after _apply_deferred_aborts
D unknown uid → idempotent ack
E duplicate abort → single free + single ack
F frontend abort-pending gates late UserReply; dup ack is a no-op
G natural finish (stop_token_ids) → UserReply only, no ack
H tokenizer AbortAckMsg → AbortAckReply forwarding
* test_scheduler_overlap_abort_fence.py fixture: added
_pending_abort_acks = [] to the Scheduler skeleton so the 2.3e
suite still runs against the updated _process_one_msg.
Verified on remote container 998ce5ba6e5e:
* 8/8 Gate 2.3f hermetic tests PASS.
* 25/25 Gate 2.3b-e hermetic tests PASS (regression clean).
* 69/69 Ascend FIA backend tests PASS.
* /tmp/gate23f_probe.py in-process end-to-end message flow: PASS
on both flow A (inflight abort → deferred ack → frontend cleaned)
and flow B (natural finish, no ack).
Deliberately out of scope: shutdown-time ack synthesis, HTTP cancel
endpoint, model.forward/FIA fault recovery, sampler-fault recovery,
overlap fence semantic changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Freezes Gate 2.3 (request lifecycle and cancel protocol on Ascend 910B1). Runtime commits d520a71 (allocation rollback), f56ce2a (sampler atomicity), 4ef0c15 (shutdown drain), ada1688 (overlap abort fence), ac1bb8e (end-to-end abort ack) are attested by 33 hermetic tests plus fresh 910B1 evidence for allocation rollback, shutdown drain, and abort-ack message flow. Sampler-atomicity and overlap-fence real-device evidence are inherited from the runtime commits and re-attested by their hermetic regressions in this session; two historical fresh-re-attestation scripts were excluded per the freeze integrity note in section 13. No runtime code was modified in Gate 2.3g.
Ray-RP
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is documentation-only. It flips the fork's public identity to "Mini-SGLang Ascend — Technical Preview" ahead of the
v0.1.0a1tag, rewrites the README to lead with the Ascend 910B1 story, and preserves the upstream CUDA/H200 README verbatim in a labeled "Upstream documentation" section.Not intended for merge into
sgl-project/mini-sglang. Filed here for visibility only — the Ascend-focused landing content, license attribution, and limitations table are specific to the downstream fork and should not overwrite upstream's own README.Public fork: https://github.com/Ray-RP/mini-sglang-ascend
What changed
README.mdrewritten (Ascend-first, upstream preserved):sgl-project/mini-sglang(MIT), technical preview, verified on 910B1 with TP=1 in eager mode.Ascend 910B1 · Qwen3-0.6B · TP=1 · eager · npu_fia · Status: validated.Limitationssection: TP>1 unverified, Qwen3-0.6B only, HTTP+ZMQ path not frozen, no soak, no perf-leadership claim, not upstream-merged.## Upstream documentation.docs/ascend_port/gate1_verdict.md: removed a real container ID from a section header (replaced with "verification container").docs/ascend_port/source_dependency_audit.md: replaced two absolute developer paths with neutral descriptions ("remote host checkout", "local development checkout").No runtime code, no dependency, no license, no CI change.
Verification
pytest -q -o addopts="" tests/misc/test_pyproject_config.py→ 14 passedMini-SGLang Ascend,Technical Preview,Ascend 910B1,TP=1,eager, upstream lineage link, MIT license link.Notes for upstream reviewers
If upstream would prefer that the Ascend port not touch the top-level README at all in this fork's PR series, this one can simply be closed without merging — everything it lands is fork-local. It's filed here purely so the upstream project has visibility into the identity / license attribution changes the fork carries.