QKVShare: Quantized KV-Cache Handoff for On-Device LLMs

What changed

Historically, enabling multi-agent LLM systems on resource-constrained edge devices faced a dilemma: how to efficiently transfer the “memory” or contextual state (the KV-cache) between different agents. The prevailing methods were either computationally expensive full re-prefill, where each new agent re-processes the entire conversation history, or transferring the KV-cache in full precision, which is memory-intensive and slow for on-device operations. Both approaches severely limited the practicality of complex, multi-hop agentic workflows on edge hardware.

QKVShare, as presented on arXiv on May 6, 2026, introduces a novel approach: quantized KV-cache handoff. This framework combines several key innovations. First, it employs token-level mixed-precision allocation, meaning different parts of the KV-cache can be quantized to varying degrees based on their importance or sensitivity. Second, it packages this quantized state into a “CacheCard” representation, designed for self-containment and efficient transfer. Finally, it provides a HuggingFace-compatible injection path, streamlining integration into existing LLM ecosystems. This combination directly addresses the trade-off between memory footprint, computational overhead, and contextual fidelity during agent handoffs [arXiv:2605.03884v1].

Prior work in KV cache quantization, such as TurboQuant and KVQuant, focused primarily on reducing the memory footprint for single-model inference, enabling larger models or longer contexts on limited RAM [6, 1]. While these techniques are crucial for on-device LLMs, they didn’t explicitly tackle the inter-agent transfer problem. QKVShare extends this by making quantized KV-caches not just smaller, but also transferable and re-injectable across agents, a critical step for true multi-agent orchestration on edge devices [4].

How it works

QKVShare’s core mechanism revolves around the efficient serialization, transfer, and deserialization of a quantized KV-cache. When an agent needs to hand off its context to another, its current KV-cache, which stores the key and value states of previous tokens, is processed. Instead of transferring the full-precision floating-point values, QKVShare applies token-level mixed-precision quantization. This means that certain tokens or parts of the cache deemed less critical for future predictions might be aggressively quantized (e.g., to 4-bit or 8-bit integers), while more critical parts retain higher precision if necessary. This adaptive approach aims to balance memory reduction with minimal accuracy degradation.

The quantized KV-cache is then encapsulated into a “CacheCard.” This CacheCard is designed to be a self-contained representation of the agent’s state, making it easy to transfer between different LLM instances or even different physical devices. Once transferred, the receiving agent uses a HuggingFace-compatible injection path to load the CacheCard directly into its KV-cache memory. This bypasses the need for the receiving agent to re-process the entire conversation history from scratch (re-prefill), which is a computationally intensive operation. The framework’s design targets a streamlined process: quantize, package, transfer, inject, and then continue generation from the injected state [arXiv:2605.03884v1].

This approach differs from distributed KV cache solutions like those found in vLLM, which focus on disaggregated serving and offloading for large-scale inference [5, 8]. QKVShare specifically targets on-device, multi-agent scenarios where direct, local transfer of context is paramount for responsiveness and privacy.

Why it matters for operators

For operators building and deploying multi-agent LLM systems on edge devices, QKVShare represents a significant step towards practical, responsive, and resource-efficient applications. Until now, the promise of sophisticated on-device agentic AI has been hampered by the sheer computational cost and memory footprint of maintaining and transferring conversational context. The traditional approaches of full re-prefill or full-precision KV-cache transfer are simply not viable for the latency and memory constraints of edge hardware.

This framework directly impacts the feasibility of deploying complex agentic workflows, such as personal assistants that delegate tasks, diagnostic systems that pass context between specialized sub-agents, or interactive gaming NPCs with deep memory. By drastically cutting down the time-to-first-token (TTFT) during agent handoffs, QKVShare enables smoother, more natural multi-turn interactions. Operators can now consider architectures where multiple smaller, specialized LLMs collaborate on a single device, rather than relying on a single, monolithic model or constant cloud round-trips. This not only improves user experience but also enhances data privacy by keeping more processing local.

However, operators should be wary of the “quantization tax.” While QKVShare shows competitive results, the paper itself notes that adaptive quantization’s gains against uniform quantization are clearest in “deeper-hop, higher budget settings.” This implies that for simpler, shallower agentic tasks, the overhead of adaptive quantization might not yield proportional benefits, or uniform quantization could be “good enough.” The critical takeaway for operators is to carefully benchmark the specific quantization strategy against their application’s requirements for accuracy and context depth. Don’t assume adaptive is always superior; sometimes, simpler is faster and sufficient. The paper also highlights that post-injection generation, not card creation, is the current bottleneck, suggesting future optimization efforts should focus on the efficiency of the model resuming generation from a quantized state. This means operators should prioritize models and inference engines that are highly optimized for quantized inference, not just efficient KV-cache management.

Benchmarks and evidence

QKVShare’s efficacy was evaluated using Llama-3.1-8B-Instruct on 150 GSM8K problems, focusing on context handoff latency and the impact of quantization on performance [arXiv:2605.03884v1].

• Handoff Latency (Time-to-First-Token, TTFT):
  • At a nominal 1K context, QKVShare reduced TTFT to 130.7 ms, compared to 150.2 ms for full re-prefill.
  • At a nominal 8K context, the reduction was more substantial, with QKVShare achieving 397.1 ms TTFT versus 1029.7 ms for full re-prefill. This represents a 63% reduction in TTFT at higher contexts.
• Quantization Performance:
  • Adaptive quantization within QKVShare remained competitive under repeated handoffs.
  • It showed its clearest gains against uniform quantization in deeper-hop, higher-budget settings, indicating its value for complex multi-agent workflows.
• Stage Timing:
  • Analysis revealed that post-injection generation, rather than the CacheCard creation itself, dominates the current QKVShare latency path. This suggests that while handoff is faster, the subsequent inference from the injected, quantized state still holds optimization potential.

These results position quantized KV handoff as a promising direction for on-device systems, particularly for reducing the initial latency when switching context between agents. For comparison, general KV cache quantization techniques like those discussed in the llama.cpp TurboQuant discussion show varying impacts on perplexity and generation speed. For instance, at a ~110K context, q4_0 quantization could lead to a 36.8% degradation in perplexity compared to f16, and generation could be 37% slower [1]. This underscores the importance of QKVShare’s adaptive quantization and its focus on maintaining performance during handoffs, rather than just raw memory reduction.

Risks and open questions

• Accuracy vs. Quantization Depth: While QKVShare’s adaptive quantization shows promise, the exact trade-off between quantization level, accuracy, and task performance for diverse multi-agent workloads remains an open question. The paper notes gains against uniform quantization are clearer in “deeper-hop, higher budget settings,” implying a potential sweet spot that needs careful tuning for different applications.
• Generalizability Across Models: The current results are based on Llama-3.1-8B-Instruct. The effectiveness and optimal quantization strategies might differ significantly for other LLM architectures or sizes. Operators will need to validate QKVShare’s performance with their specific models.
• Post-Injection Generation Bottleneck: The finding that post-injection generation dominates the latency path suggests that while the handoff itself is faster, the subsequent inference from a quantized and injected state still has room for optimization. This could involve further advancements in quantized inference kernels or specific model fine-tuning for quantized KV-cache use.
• Controller Ablations and Runtime Comparisons: The authors highlight the need for “stronger controller ablations and apples-to-apples runtime comparisons.” This indicates that a more comprehensive understanding of how different multi-agent orchestration strategies interact with QKVShare, and how its end-to-end performance compares to other emerging solutions, is still required.
• Integration Complexity: While a HuggingFace-compatible injection path is a good start, integrating this framework into production multi-agent systems might still present engineering challenges, especially for custom agent frameworks or highly optimized inference stacks.