Can we compress (encode) sequence of token vectors and do attention with compressed versions

[vukrosic]

This needs to be though about further and scoped out

[MRiabov]

After your last video: how about this:

1. Chunk text,
2. classify the chunks into groups,
3. Compress (embed or summarize) each group's information using a dedicated compression module (in essence, an "expert" as in MoE.)
4. Generate the final output using summaries from each group.

The idea is to have different experts compress different types of info. Running each expert over its data is cheaper than running the full model - in particular cheaper as in: O(N^2) vs O((N/num_experts)^2+(len_of_output_per_experts*num_experts)^2). Using reinforcement learning then make the experts cooperate such that each info is provided is useful.

Here, let's take sample values:
1M tokens, num_experts=12, max len of output per expert=10k tokens.
Then the final output would be computed over 120k tokens and not 1M tokens, which is useful. Of course, there is higher computational cost+latency, but latency may be solved with using diffusion as summarizers since they are much faster?

So, to recap: chunk-> group -> compress using experts -> use compressed output for generation.

I wonder whether sparse attention hasn't solved this already. Why bother with compressed values if sparse attention exists?

[MRiabov]

So to clarify: frame encoding process as a separate MoE problem alongside the already existing MoE decoder.

[nofreewill42]

I've tried something similar in the past:
qwen2.5 QLoRA fine-tuning on a task to summarize a given text into embedding(s) so that looking only on the embeddings, the model has to reconstruct the context from these embeddings

A concrete example to what, or how I did the model training:
Let "I'm an electrical engineer" be the text to be embeded into an "embedder token" and then decode the same input context.

"I'm an electrical engineer<|context_extractor_token|>I'm an electrical engineer"

Some results:
Input: The computer is below that table.<|context_extractor_token|>
Decode: <|context_extractor_token|>The computer is under the table.
Input: The lamp, up there on the ceiling, is emitting red light.<|context_extractor_token|>
Decode: <|context_extractor_token|>The light from the lamp is red.
Input: Qwen2.5 is an LLM from Alibaba and it's awesome!<|context_extractor_token|>
Decode: <|context_extractor_token|>Awesome! AI is amazing and awesome.
Input: Mistral was good but it's old now.<|context_extractor_token|>
Decode: <|context_extractor_token|>Mistral is old and not good anymore.

More here in an X post

And the junky training implementation is here

[MRiabov]

I've tried something similar in the past: qwen2.5 QLoRA fine-tuning on a task to summarize a given text into embedding(s) so that looking only on the embeddings, the model has to reconstruct the context from these embeddings

A concrete example to what, or how I did the model training: Let "I'm an electrical engineer" be the text to be embeded into an "embedder token" and then decode the same input context.

"I'm an electrical engineer<|context_extractor_token|>I'm an electrical engineer"

Some results: Input: The computer is below that table.<|context_extractor_token|> Decode: <|context_extractor_token|>The computer is under the table. Input: The lamp, up there on the ceiling, is emitting red light.<|context_extractor_token|> Decode: <|context_extractor_token|>The light from the lamp is red. Input: Qwen2.5 is an LLM from Alibaba and it's awesome!<|context_extractor_token|> Decode: <|context_extractor_token|>Awesome! AI is amazing and awesome. Input: Mistral was good but it's old now.<|context_extractor_token|> Decode: <|context_extractor_token|>Mistral is old and not good anymore.

More here in an X post

And the junky training implementation is here

hey, that's a good post/demo. However I recall that e.g. recent work by cognition (now owners of Windsurf) shows that embedding-based retrieval is not that precise.

You should try testing it on some more serious benches like U-math

[nofreewill42]

This does not really have anything to do with retrieval - although this approach might help it.

An LLM just takes an imput context, a "compressor" token looks at it through the LLM, and then the same model, given the compressor token only, needs to reconstruct the context that created the compressor token.

I imagine that if we used something like this for a long context chunked and compressed this way sentence-by-sentence, if the compressor tokens could attend to past compressor tokens as well, then the reconstruction could be much better. But this is only my hypothesis, not tested.

The approach could help retrieval by forcing the embeddings to really "contain" the information being embedded.

[MRiabov]

if the compressor tokens could attend to past compressor tokens as well

Latency would be high for such compression, just saying. What you are describing is in effect an RNN with a sliding window approach.

[nofreewill42]

I don't see how latency would be higher - it is indeed lower compared to token-based LLMs, vs. this "sentence"-based LLM.
And I don't see how it is in effect an RNN - how an LLM with a sliding window is in effect an RNN?

[MRiabov]

I don't see how latency would be higher - it is indeed lower compared to token-based LLMs, vs. this "sentence"-based LLM. And I don't see how it is in effect an RNN - how an LLM with a sliding window is in effect an RNN?

The latency to the first token can be higher because you want to preprocess sequentially:
"if the compressor tokens could attend to past compressor tokens as well, then the reconstruction could be much better. ".

Perhaps, this should be done not only to past tokens, but to the nearest context. I.e. chunks with overlap, but the overlap is over more sentences.

[nofreewill42]

All the sentence embeddings can be created parallely, regardless of the other sentences (with a block diagonal causal mask).
Then we can extract all these sentence embeddings and do attention on them with a causal mask, and then predict the texts from each of these embeddings enriched with all the past sentence embeddings.
No need to wait for the first sentence embedding to finish to start creating the second embedding. We create all of them separately, and then we enrich them with each other.

[MRiabov]

and there we go: https://huggingface.co/papers/2510.25741

[nofreewill42]

and here we go as well:

"CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9% accuracy."

[MRiabov]

and here we go as well:

"CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9% accuracy."

I think I'll try training a looped reasoning model + compression - the paper that you've linked. Want to join?

[nofreewill42]

I see you are from Ireland. I'm from Hungary.. Yesterday's match was insane, congrats! ;)
And yeah, sounds like a cool project, I would like to join.
Shall we connect somewhere for discussion?

[MRiabov]

I see you are from Ireland. I'm from Hungary.. Yesterday's match was insane, congrats! ;) And yeah, sounds like a cool project, I would like to join. Shall we connect somewhere for discussion?

Whatsapp +353 eighty-nine (sorry, just preventing against scrapers/spammers), 948 5670

[MRiabov]

So the idea is essentially: use the transformer input/output heads as in CALM that you've linked that outputs values from looped reasoning in the LoopLM.

Two papers from the above:

https://arxiv.org/pdf/2510.27688
https://arxiv.org/pdf/2510.25741#page=2.93