Skip to content

README.md

Latest commit

 

History

History
373 lines (286 loc) · 12.4 KB

README.md

File metadata and controls

373 lines (286 loc) · 12.4 KB
Raw

Hidden Decoding at Scale

Latent Computation Scaling for Large Language Models

WeChat AI Team, Tencent

Paper PDF   Blog   Models   License


Same Transformer backbone, more latent computation per token.

This repository hosts the Hidden Decoding paper, public demonstration checkpoints, and inference patch.

Updates

  • 2026-07-07: Added the paper PDF, frontier-scale WeLM-HD4-80B/617B results, and the Stream-Factorized Attention framing.
  • 2026-04-01: Released Hidden-Decoding-8B-n8-Instruct, our first instruction-tuned demonstration model.
  • 2026-03-10: Initial release of base models (n=2, n=4, n=8) and SGLang inference patch.

Paper

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

The paper studies whether an already-strong LLM can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Hidden Decoding expands each token into multiple hidden streams, supervises only the final stream, and retains the intermediate streams' KV cache so their latent computation remains available to later tokens.

Key Idea

Hidden Decoding scales computation along the sequence-length dimension:

  1. Each input token is expanded into n streams with independent embedding tables.
  2. The expanded length-nL sequence is processed by the same Transformer backbone.
  3. Only the final stream predicts the next token.
  4. Intermediate streams act as latent computation states and keep their own KV cache.

This differs from recurrent-depth or looped Transformers: the extra computation is a longer sequence in a single forward pass, which fits the pipeline-parallel training stack used for large MoE models.

To make the expanded sequence trainable at scale, the paper introduces Stream-Factorized Attention. Most layers attend only within each stream, and only a subset of layers mix information across streams. This reduces the attention growth from quadratic in n to roughly linear in n.

Main Results

Frontier-Scale MoE

We train WeLM-HD4-80B and WeLM-HD4-617B with n=4 during continued pretraining. The matched autoregressive and Hidden Decoding models use the same early SFT-only post-training recipe, with no reinforcement learning. Active Transformer parameters per token stay unchanged: 3B for 80B and 23B for 617B.

Benchmark WeLM 80B WeLM-HD4 80B Delta WeLM 617B WeLM-HD4 617B Delta
GPQA Diamond 87.6 88.8 +1.2 89.1 91.2 +2.1
HLE 27.4 28.4 +1.0 33.6 35.4 +1.8
MMMLU 84.4 85.6 +1.2 86.4 87.5 +1.1
FrontierMath 45.8 49.0 +3.2 49.0 51.0 +2.0
PHYBench 69.8 73.8 +4.0 75.3 76.3 +1.0
MathArena Apex 16.4 20.1 +3.7 24.2 24.7 +0.5
HMMT 93.3 94.1 +0.8 96.0 96.2 +0.2
IMO-AnswerBench 85.0 85.3 +0.3 87.5 88.5 +1.0
SciCode 45.8 50.0 +4.2 51.4 52.1 +0.7

The 4x expanded sequence costs 5.1x per batch on WeLM-HD4-80B and 4.4x per batch on WeLM-HD4-617B, close to the 4x linear reference and far below the 16x dense-attention baseline.

Expansion-Factor Scaling

Increasing the expansion factor improves language modeling loss and downstream accuracy while keeping the Transformer backbone fixed.

Model / Metric Base n=2 n=4 n=8
80B MoE Pile-test BPB (lower is better) 0.386 0.387 0.382 0.378
80B MoE MMLU 85.1 85.0 86.7 87.5
80B MoE BBH 87.5 88.3 90.0 90.6
Qwen3-8B MMLU 79.8 80.9 81.9 82.2
Qwen3-8B BBH 78.8 81.3 83.0 83.9
Qwen3-8B MATH 56.0 58.2 60.0 61.1

Released Demonstration Models

The released HuggingFace checkpoints are not the main models of the paper. They are Qwen3-8B based demonstration models provided to show that Hidden Decoding can scale with the expansion factor n: as n increases, the same Transformer backbone receives more latent computation per token and generally improves. The main paper results are the WeLM-HD4-80B and WeLM-HD4-617B studies above.

Base Models

All released base models share the same 8B Transformer backbone. They differ by the Hidden Decoding expansion factor n.

Model Scale Training Tokens Link
Hidden-Decoding-8B-n2 2x 75B HuggingFace
Hidden-Decoding-8B-n4 4x 150B HuggingFace
Hidden-Decoding-8B-n8 8x 187B HuggingFace

Instruct Model

Model Scale Base Model Link
Hidden-Decoding-8B-n8-Instruct 8x Hidden-Decoding-8B-n8 HuggingFace

Qwen3-8B Results

These released 8B models are included as a public, reproducible demonstration of expansion-factor scaling. They are not intended to be the paper's main model release.

Base Model

Evaluated on Qwen3-8B-Base with progressive Hidden Decoding scaling:

Benchmark # Shots 8B Baseline 8B scale n=2 8B scale n=4 8B scale n=8
BBH (EM) 3-shot 78.8 81.3 83.0 83.9
MMLU (EM) 5-shot 79.8 80.9 81.9 82.2
MBPP+ (Pass@1) 1-shot 66.7 69.4 68.7 69.4
MATH (LLM-judge) 4-shot 56.0 58.2 60.0 61.1
ARC-C 25-shot 93.9 94.3 94.4 94.7
HellaSwag 10-shot 79.7 83.1 85.0 85.3
GSM8K 4-shot 92.5 93.3 93.9 94.6

Instruct Model

Instruction-tuned from Hidden-Decoding-8B-n8 and compared with Qwen3-8B-Instruct plus a matched Qwen3-8B SFT baseline trained on the same data:

Benchmark Category Qwen3-8B-Instruct Qwen3-8B SFT Hidden-Decoding n=8 SFT
AIME24 Math 29.42 19.09 30.00
AIME25 Math 20.58 17.67 26.25
MATH500 Math 85.16 84.19 89.04
GPQA Diamond Reasoning 48.79 50.91 55.25
CMMLU Knowledge 78.20 75.05 81.15
C-Eval Knowledge 78.67 76.04 83.20
MMLU-Pro Knowledge 65.84 68.35 75.70
SuperGPQA Knowledge 36.79 41.20 48.35

Sampling: temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5, max_tokens=4096. Judge: GPT-4o.

Installation (Inference)

Option 1: Docker Image

docker pull aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126

Option 2: Use the Forked SGLang

git clone https://github.com/exlaw/sglang.git
cd sglang
pip install -e "python[all]"

Option 3: Apply Patch Manually

Apply the patch on top of SGLang:

git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout 4efe844a2
git apply /path/to/hidden_decoding.patch
pip install -e "python[all]"

Serving

Launch the server:

python -m sglang.launch_server \
    --model-path tencent/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128
Docker 
docker run --gpus all -p 8080:8080 -p 8081:8081 \
  -v /path/to/models:/models \
  aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126 \
  bash -c "python -m sglang.launch_server \
    --model-path /models/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128"

Hidden Decoding models process n times longer sequences internally, so --chunked-prefill-size -1, --attention-backend fa3, and conservative batch sizes are important for stability and performance. Adjust --tp-size for multi-GPU setups.

Usage

Chat Completions

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain hidden decoding in simple terms."},
    ],
    max_tokens=512,
    temperature=0.7,
)
print(response.choices[0].message.content)

Text Completions

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
response = client.completions.create(
    model="default",
    prompt="The meaning of life is",
    max_tokens=128,
    temperature=0,
)
print(response.choices[0].text)

Chat UI

We also provide a lightweight, zero-dependency web chat interface for interactive testing:

python chat_ui.py

Then open http://localhost:8081 in your browser. The Chat UI connects to http://localhost:8080/v1 by default.

Docker (serve model + Chat UI together) 
docker run --gpus all -p 8080:8080 -p 8081:8081 \
  -v /path/to/models:/models \
  -v /path/to/chat_ui.py:/app/chat_ui.py \
  aiweiliu/sglang-scale-seq:v0.5.2rc2-cu126 \
  bash -c "python -m sglang.launch_server \
    --model-path /models/Hidden-Decoding-8B-n8-Instruct \
    --trust-remote-code \
    --tp-size 1 \
    --port 8080 --host 0.0.0.0 \
    --chunked-prefill-size -1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.82 \
    --max-running-requests 32 \
    --context-length 131072 \
    --cuda-graph-max-bs 128 \
    --cuda-graph-bs 1 2 4 8 16 32 64 128 &
  sleep 5 && python /app/chat_ui.py"

Features:

  • Streaming responses with real-time token speed display
  • Thinking/reasoning block visualization
  • Configurable system prompt, temperature, and max tokens
  • Pure Python standard library implementation

Patch Contents

The patch adds the qwen3_scale_seq model architecture and modifies the scheduler, batch manager, and CUDA graph runner to handle expanded sequence lengths.

Citation

If you find this work useful, please cite:

@article{hidden_decoding_at_scale_2026,
  title  = {Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models},
  author = {WeChat AI Team},
  year   = {2026},
  url    = {https://github.com/Tencent/Sequential-Hidden-Decoding/blob/main/paper/hidden_decoding_at_scale.pdf}
}

Contact

Sijun Zhang (nepheloturbulence@gmail.com), Aiwei Liu (liuaiwei20@gmail.com)

License

This project is released under the License Terms of Hidden-Decoding. The dependent open-source models and software components remain licensed under their respective original licenses. See LICENSE for details.