LastKV-LLM-8B-Base
Final model checkpoints for LastKV: Last Layer KV Cache is All You Need.
| Training stage | Final step | Configured context | Revision |
|---|---|---|---|
| Continued pretraining, including the learning-rate cooldown | 59,904 | 8,192 tokens | main, also pinned as 8k-step-059904 |
| Subsequent long-context training | 1,209 | 262,144 tokens | 256k-step-001209 |
Each checkpoint contains the raw model weights consolidated from its distributed training checkpoint: 8,030,269,952 parameters in 419 tensors, stored as four BF16 safetensors files (about 16 GB total), with the model configuration and tokenizer. The index combines the four files into one model checkpoint. Optimizer and dataloader states are excluded.
Download
from huggingface_hub import snapshot_download
model_path = snapshot_download(
"ZefanCai/LastKV-LLM-8B-Base",
revision="256k-step-001209", # Use "main" for the 8K pretraining checkpoint.
)
Inference requires the matching CustomRecurrentMarinAKV254ReuseKVForCausalLM implementation, which is not bundled with this weights release. The tokenizer metadata is preserved from the training export; its model_max_length can differ from the model configuration's context length.
License: MIT.