Project · 2026
PriorityKV
Compressing an agent's KV cache by what each message is, not only by where attention falls.
- ModelsQwen3-8B, Llama-3.1-8B, Qwen3-32B
- Scale18,240 generations
- WithArush Sharma
Problem
An agent's KV cache grows with every turn. Existing methods decide what to drop from attention scores, but the transcript already says what matters: the system prompt, the tool schemas and the constraints.
The question is how far that free signal goes, and when it stops being enough.
What we built
- Token roles from the chat messages, grouped into 16-token pages, under a fixed memory budget.
- Important pages stay in BF16 and the rest drop to INT4, with a FlashInfer decode path.
- ADAPT, a parameter-free blend of structure and attention that leans on attention as the budget gets tighter.
- PriorityBench-A, a benchmark of synthetic agent traces, and a sweep across three models.
Design choices
- Eviction quality, INT4 quality and system cost are measured as separate questions.
- The prediction for the external benchmark (BFCL) was written down before scoring.
Result
Structure-aware retention matches attention-based methods such as SnapKV at a 25% budget, and ADAPT never loses to SnapKV across 48 settings.
Tool schemas stay fully intact even when the budget is 18 times oversubscribed. Peak memory drops to 0.87×.