server : host-memory prompt caching

[ggerganov]

target #16440
rel #16117

Initial version of automatic memory offloading to host memory using an extended logic for minimizing the prompt reprocessing. The host-memory prompt cache acts as "extra slots" with which we can calculate prefix similarity and decide to hot-swap them into the llama_context if it would reduce the processing.

Still WIP, but probably should be useable already.

Note: mtmd workarounds are starting to cause some headaches. For example server_tokens is not copyable which complicates the cache logic and makes the prompt caching feature incompatible with mtmd.

Server refactor

• Replace server_slot members with a single server_task
• Remove server_slot.n_predict
• Remove prompt truncation logic (obsolete and not useful anymore)
• slot.task is now const ptr to reflect that the task parameters should not change when it is passed to the slot

TODOs

• Set memory limit for the host-memory cache from CLI
• Clean-up implementation
• Test with agentic workflows
• Multi-slot tests
• Fix progress report

[ggerganov]

Looking for some feedback of how this new logic performs in different use cases. I've been testing it with the llama.vscode agent and it significantly improves the experience since we can now use a single server slot without trashing the prompt cache.

The current implementation should work with any model (dense, MoE, SWA, SSM, etc.). I think the default settings should be good for most use cases, though we'll probably add some options to adjust cache limits if needed.

Pay attention to these new messages in the logs:

Interested in testing agentic use cases, such as Claude Code and similar, where we have a single large context with various auxilary calls (keyword extraction, summarization, etc.) interleaved. The expectation is that prompt reprocessing should be significantly reduces in such cases.