Feature Request: Support for Activated LoRA

[kgreenewald]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Activated LoRA (aLoRA) is a new family of LoRA adapters that are invoked by including an invocation string in the prompt, and the weights are only adapted for the tokens in the sequence after the invocation prompt appears (think of this as an "adapter generation prompt"). All prior context tokens have unadapted weights in the transformer, allowing the inference engine to utilize any existing KV cache from prior base model inference calls. This means that one can apply the aLoRA adapter deep in a multi-turn interaction with the model without needing to recompute the entire KV cache. Instead, the adapter can use the KV cache from the base model right up until the adapter is invoked, thus significantly reducing TTFT.

This issue proposes supporting inference for aLoRA adapters trained with (or at least following the same format as) Huggingface PEFT (active PR). It extends work building out support in vLLM (active PR).

paper: https://arxiv.org/abs/2504.12397
blog: https://research.ibm.com/blog/inference-friendly-aloras-lora
Huggingface PEFT PR: huggingface/peft#2609
vLLM PR: vllm-project/vllm#19710

results from paper:

Motivation

Activated LoRA enables up to 10-30X faster inference than LoRA when using adapters for specialized turns in a multi-turn LLM use case with prefix caching. This is particularly relevant for agentic (including RAG) applications (see the paper for extensive examples and further discussion of use cases), enabling weight-tuning of key skills with (almost) no inference latency downsides relative to prompt optimization.

Inference support in llama.cpp is important to enable local inference for aLoRA adapters (typically trained using the HF PEFT PR), complementing the vLLM inference support in progress.

Possible Implementation

Modify the current support for hot-swapping LoRA adapters to recognize Activated LoRA adapters. Adjust the inference code for the adapter to activate weights only after indicated by the invocation prompt. Ensure that prefix caching correctly recognizes which substrings used which weights (base model or adapter weights).

[kgreenewald]

See discussion #15213

[kgreenewald]

cc @gabe-l-hart

[ggerganov]

I don't understand why this approach would improve the performance of llama.cpp. Hot swapping a LoRA has negligible overhead and we already have the KV cache stored for the entire conversation.

[kgreenewald]

I don't understand why this approach would improve the performance of llama.cpp. Hot swapping a LoRA has negligible overhead and we already have the KV cache stored for the entire conversation.

Thanks for replying! There's a key point that perhaps we did not make clear. First, just so we are on the same page (a) this feature would not change the overhead of swapping (b) we will not change KV cache storing.

Instead, the purpose of Activated LoRA is to make the context KV cache of the adapter interchangeable with the KV cache of the base model. This is not possible with standard LoRA, because the weights have changed.

Why this matters - this enables efficient (i.e. no need to do long prefill operations) when switching between base model and any aLoRA adapters in a multi-turn "conversation". Normally, in such cases LoRA - since it has different weights - will have to prefill a long history before it can generate, since this history has been created and/or input to the base model. Activated LoRA removes this friction point by directly using the pre-existing KVs of the base model. See the paper and issue for more details, or happy to elaborate further here!

[gabe-l-hart]

Thanks for writing this up @kgreenewald! @ggerganov, hopefully Kristjan's comments helps clarify the value. What helped me understand when he first showed it to me was a concrete example:

Consider a multi-turn RAG scenario with a retrieval engine. Several turns in, the user asks a question that relies on the previous messages in the conversation for context (eg "tell me more about that" referencing previous discussion of "widgets"). The RAG system needs to formulate a query that is self contained. This could be done by using the entire conversation history as the query (with obvious drawbacks) or the system could itself rewrite the context into a concise self-contained query (eg "detailed information about widgets").

This query rewrite could be done with prompt engineering or with a LoRA adapter. If using a standard adapter, the entire context would need to be re-prefilled. With an aLoRA, it would not and could simply reuse existing cache.

The benefit gets further extended the more sub-tasks you add to the system. This same system could use an aLoRA for citation generation, question answerability evaluation, groundedness evaluation on the answer, etc. Without aLoRA, these would each be a fresh pass through prefill. With aLoRA (and adapter hot-swapping), each sub-task would reuse existing cache.

[ggerganov]

What changes do you think are needed in llama.cpp to support aLoRA?

I could still be missing something, but from the conversation and reading the paper briefly, I think all of this is already supported - just hotswap the LoRA after the prefill is done.

[gabe-l-hart]

Interesting! You may well be right that it already just works. I had assumed that when activating the adapter, the cache would be invalidated, but if that's not the case, then this might be a no-op. I haven't actually unboxed the code thoroughly yet, so will need to dig a bit. I'll report back here and we can close out if it's already good to go.

[ggerganov]

The KV cache won't be cleared implicitly when you apply a LoRA - the user is responsible to do that in case they want to. In this case, there is no need to clear it, so you can simply proceed from the existing state - I think this is what you need.

[gabe-l-hart]

Awesome. In that case, there might be some changes to allow this cleanly through the server APIs, but it sounds like the underlying implementation is already there.

[ggerganov]

Yes, on the llama-server level likely you just need to adjust this logic here:

llama.cpp/tools/server/server.cpp

Lines 2248 to 2254 in c24f4e2

	if (!are_lora_equal(slot.params.lora, slot.lora)) {
	// if lora is changed, we cannot reuse cached tokens
	slot.cache_tokens.clear();
	slot.lora = slot.params.lora;
	}

[gabe-l-hart]

Very nice. I'll dig there

[gabe-l-hart]

After digging a bit deeper, I have a few thoughts on implementation to discuss:

1. It looks like the full implementation is still pending review/merge in PEFT. In particular, the adapter_config.json field used to distinguish alora from lora is in flux, so there is a discrepancy between the example alora configs and the current planned implementation (something @kgreenewald is fixing I believe). To me, this means we'll need to hold off on an merging any implementation here until the upstream work is finalized.

2. It looks like the mechanism for determining when, and where to apply the alora is a little bit more involved than I had previously thought. My assumption was that this determination would be made on a per-request basis such that if a request were given with the alora, it would simply toggle it on for all "new" tokens which, when thinking more about it, is not something a stateless server would be able to determine without inspecting its own cache. The mechanism in the PEFT and vLLM implementations for detecting when/where to activate the alora is to identify the presence of an activation sequence in the tokenized representation and apply the alora to all tokens after the last instance of the activation sequence.

As a result of (2), I think the implementation here may be a bit deeper than I had previously hoped. I've got a preliminary branch (https://github.com/gabe-l-hart/llama.cpp/tree/gabe-l-hart/alora-support) that has all of the hparam plumbing and parsing, but the actual usage implementation will likely involve implementing some sort of positional mask that can be used in build_lora_mm and adjacent graph methods.

[kgreenewald]

Yes, on the llama-server level likely you just need to adjust this logic here:

llama.cpp/tools/server/server.cpp

Lines 2248 to 2254 in c24f4e2

if (!are_lora_equal(slot.params.lora, slot.lora)) {
// if lora is changed, we cannot reuse cached tokens
slot.cache_tokens.clear();
slot.lora = slot.params.lora;
}

Great observation, and this is how we initially got some level of support on a fork of vLLM. Specifically, given an aLoRA request, you can

1. grab whatever base model cache is currently available,
2. do a dummy 1-token generation so that all remaining necessary context is prefilled into the base-model cache,
3. change the logic (similar to how you suggest) so that the alora grabs this base-model cache.

There is an inefficiency here, specifically, doing the dummy token generation prior to the real generation creates 2 forward passes through the model where 1 would have sufficed. For this reason, plus tricky aspects of the cache block system on vLLM, we ended up having to do a deeper modification to how the lora layers were implemented to explicitly adapt only the most recent chunk of tokens.

If we were to go this "hacky" route with a dummy call, I think this would still require some modifications to the server code, something like

1. logic for reading in the adapter file to get the invocation_tokens
2. logic for each request that comes in to track where in the input string the weights should be activated
3. something to do the resulting dummy calls
4. kv cache logic change as you outline.

Everything here except (3) would also have to happen for a deeper implementation. But this could be a pretty non-intrusive way of getting some initial level of support.