RAZOR gate for Qwen3-1.7B

A learned KV-cache forget gate for Qwen/Qwen3-1.7B. For every token, as it is written to the cache, it gives one keep score per KV head. Tokens scored below the threshold are evicted. The base model is not modified.

• Architecture: one MLP per layer on the layer's hidden state (2048 -> 512 -> 8 KV heads), 28 layers, 29.5M parameters (1.7% of the base model). Same architecture as kvpress's KVzapModel.
• Training: KL divergence to the uncompressed model's next-token distribution, hard keep/drop decisions with straight-through gradients, and a dual-ascent price that holds the keep rate at 30%. 39 minutes on one T4, documents of at most 1k tokens (Nemotron data, the same source as KVzap). This is seed 0. train_log.txt has the run.
• Inference: keep a token if its score is above 0 (DMS-style threshold with a 128-token sliding window).

Code, training and evaluation: https://github.com/Manaslu-AI/razor. Paper: razor.pdf in that repo (arXiv version pending).

Results

All on 16 GB T4s, kvpress RULER string_match average.

The KVzap-objective gate has the same architecture, data and hyperparameters, trained to regress KVzip+ scores. It is our own training, not an NVIDIA checkpoint. KVzip+ needs a second pass over the context.

Limits: weak on RULER common-word extraction (cwe 57 vs 79 for KVzip+ at 4k). MATH n=30 shows no loss, not a gain. Eviction under thresholding varies by input (71-85% on RULER-4k, 5th-95th percentile).

Use

pip install torch 'transformers>=4.56,<5.3' kvpress
git clone https://github.com/Manaslu-AI/razor && cd razor

from kvpress import KVPressTextGenerationPipeline  # noqa: F401, registers the pipeline
from transformers import pipeline

from razor_kv.press import LocalGatePress
from razor_kv.ragged import RaggedCache, RaggedDMSPress, enable

pipe = pipeline("kv-press-text-generation", model="Qwen/Qwen3-1.7B", dtype="float16", device_map="cuda")
enable(pipe.model)  # per-head ragged attention
press = RaggedDMSPress(LocalGatePress(path="Manalsu/razor-gate-Qwen3-1.7B"), threshold=0.0)
print(pipe(context, question="...", press=press, cache=RaggedCache())["answer"])  # evicted KV is freed, not masked

RaggedDMSPress deletes evicted entries per head and supports batch size 1. DMSPress(LocalGatePress(...)) from kvpress makes the same decisions with masking.

Citation

@misc{rawal2026razor,
  title  = {RAZOR: Learning What to Forget in the KV Cache with a Description-Length Price},
  author = {Rawal, Himansu},
  year   = {2026},
  note   = {Manaslu Labs. arXiv version pending},
  url    = {https://github.com/Manaslu-AI/razor}
}