Add ABot-World - #14539
Open
yiyixuxu wants to merge 5 commits into
Open
Conversation
…-5B) Integrates https://github.com/amap-cvlab/ABot-World (acvlab/ABot-World-0-5B-LF, Apache-2.0) as a modular pipeline. - ABotWorldTransformer3DModel: block-causal rollout over a rolling K/V cache (window eviction, pinned reference-token prefix, relative RoPE with periodic rebase), per-frame timesteps, keyboard-action adapter added onto the patch tokens, per-stream cross-attention cache. Bit-exact vs the reference (CPU/fp32) incl. eviction, rebase, and the real 5B weights. - scripts/convert_abot_world_to_diffusers.py: pure key renames, no surgery. VAE and umt5 are byte-identical to Wan-AI/Wan2.2-TI2V-5B and are reused from Wan-AI/Wan2.2-TI2V-5B-Diffusers; FlowMatchEulerDiscreteScheduler(shift=5.0) covers the warped DMD grid (scale_noise == the reference re-noise). - modular_pipelines/abot_world: text/image/reference encoders -> core denoise (prepare + rollout IterativePipelineBlocks over blocks k, with a nested distilled denoise loop over (i, t) and a KV-cache context update) -> decode. Streams via pipe.stream() (events per denoise step and per ~1s block) and drives interactively via loop_step, writing new actions into the state between calls. - Converted checkpoint + runnable example: YiYiXu/ABot-World-0-5B-LF-Diffusers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
models
size/L
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
- ABotWorldStreamingBlocks: like the default blockset, but each block is decoded to pixels inside the rollout loop (trailing-window decode with the previous block as temporal context; independent per-block decode of the causal VAE produces artifacts), so stream() yields ready frames per ~1 s block - the rollout takes a scripted `actions` list or an `action_source` callable polled once per block (return None to stop) — a live driver is just a stream() consumer whose callable returns the player's current action; with __call__ the callable combination raises, since no frames flow back between polls - set_action reads the single current `action` from state (no actions[k] indexing); `actions` becomes optional in the prepare step Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Without reference images the ref encoder emits zero reference latents with a zero reference_mask, matching the reference implementation's ref-less mode; the mask is plumbed through both transformer calls in the streaming rollout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the ABot-World streaming rollout decode(z, cache=cache) keeps the causal-conv feature cache across calls so a video can be decoded chunk by chunk with results identical to a single decode. The streaming decode step carries the cache through the rollout loop instead of re-decoding a trailing window of the previous block (bit-exact vs. a full decode; 3.3 -> 2.3 s per block on H100 at 704x1280). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ock in both presets AutoencoderTinyVideo ports madebyollin's TAEHV family as one class (config per checkpoint; defaults are taew2_2 for the Wan 2.2 latent space) with a TinyVideoDecodeCache for chunk-by-chunk decode, plus scripts/convert_taehv_to_diffusers.py and a docs page. Parity against the reference: decode 1.5e-8, streaming 2.5e-6, encode exact. The ABot-World presets now mirror the reference's vae_type switch: ABotWorldBlocks decodes every block inside the rollout loop with the full Wan VAE (ABotWorldDecodeStep + WanDecodeCache, as scripts/inference.py does), ABotWorldStreamingBlocks with the tiny VAE (ABotWorldTinyDecodeStep, as the interactive space does). The one-shot end decode is removed; it mirrored an entry point the reference never uses and needed all latents decoded at once. Streaming block time on H100: 2.3 s -> 0.9 s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
the
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://huggingface.co/YiYiXu/ABot-World-0-5B-LF-Diffusers