Kimi K3 300M — From Scratch
An independently implemented, 294.7M-parameter Kimi K3-inspired language model, pretrained from scratch on approximately 6 billion FineWeb-Edu tokens.
This is the released inference checkpoint for the Kimi K3 from Scratch research engineering project. Its architecture, training pipeline, checkpoint recovery, and implementation tests are documented in the project documentation.
Important: This is a reduced-scale, research-oriented base language model, not the official Moonshot AI Kimi K3 model, not a fine-tune of an existing Kimi checkpoint, and not an instruction-tuned/chat model. It is not affiliated with or endorsed by Moonshot AI.
Model details
| Property | Value |
|---|---|
| Total parameters | 294,676,808 (~294.7M) |
| Active parameters per token | ~100.4M |
| Backbone | 13-layer causal hybrid decoder |
| Attention | 9 Kimi Delta Attention (KDA) + 4 Gated Multi-head Latent Attention (MLA) layers |
| Attention residual | Block Attention Residuals (block size 2) |
| Feed-forward | 1 dense SiTU-GLU layer + 12 Stable LatentMoE layers |
| MoE | 64 routed experts, top-2 selected per token, plus 2 shared experts |
| Vocabulary | 49,152 tokens; tied input/output embeddings |
| Tokenizer | HuggingFaceTB/SmolLM2-135M tokenizer |
| Pretraining sequence length | 1,024 tokens |
| Configured maximum sequence length | 2,048 tokens (not a claim of evaluated 2,048-token quality) |
| Precision of released weights | FP32 |
The model uses its own PyTorch implementation, including KDA, Gated MLA, Stable LatentMoE, and attention residuals. It is not yet compatible with transformers.AutoModelForCausalLM.from_pretrained().
Training
The model was pretrained from scratch using the sample-10BT subset of FineWeb-Edu, tokenized with SmolLM2's tokenizer. Training examples are causal language-modeling token windows; no instruction fine-tuning or alignment stage was performed.
| Setting | Value |
|---|---|
| Training tokens | 6,000,017,408 |
| Optimizer updates | 183,106 |
| Effective tokens per update | 32,768 (32 × 1,024) |
| GPU used for the full training run | NVIDIA H100 NVL |
| Training precision | BF16 mixed precision |
| Optimization | Muon + AdamW |
| Peak learning rate | 0.003 |
| LR schedule | Warmup (1,832 updates) followed by cosine decay |
| Weight decay | 0.1 |
| Final validation loss | 3.1825 (cross-entropy / negative log-likelihood, in nats per token) |
The final validation number is an internal sampled validation measurement from the training run (8 batches of 32 × 1,024 tokens), not a standardized external benchmark. See the training and results documentation for the methodology, monitoring, and limitations. No general downstream benchmark performance claim is made here.
Independent next-token evaluation
A separate evaluation scored 65,536 validation token targets (64 fixed, non-overlapping windows of 1,024 targets), outside the short prefix repeatedly monitored during training.
| Metric | Result |
|---|---|
| Mean negative log-likelihood | 3.0510 nats/token |
| Perplexity | 21.14 |
| Top-1 next-token accuracy | 42.27% |
These are measurements for this particular prepared validation split and evaluation procedure, not a standard external benchmark. They should not be directly compared with the training monitor loss: the data windows and execution conditions differ. The model has not been instruction-tuned.
Quick Start
The pretrained weights are available on Hugging Face. Inference uses the custom PyTorch implementation from the accompanying GitHub repository.
Requirements: Python 3.12+ and uv.
git clone https://github.com/adriaflores/kimi-k3-from-scratch.git
cd kimi-k3-from-scratch
uv sync
Run greedy text generation:
uv run --with safetensors python scripts/generate_hf.py \
--source adriaflores/kimi-k3-300m-from-scratch \
--prompt "The capital of France is" \
--max-new-tokens 4
The model weights and tokenizer are downloaded automatically from Hugging Face.
The validated reference inference path uses CPU, FP32, and chunkwise KDA. Generation recomputes the full context at each step and is intended for reproducibility rather than optimized serving.
Note: This model uses a custom architecture and cannot currently be loaded directly with AutoModelForCausalLM.from_pretrained().
Verification and limitations
- The published
model.safetensorsfile was downloaded independently and its SHA-256 matched the exported local file. - Reconstructed safetensors weights and the final trainer checkpoint produced identical next-token logits on a 64-token test input, on CPU with FP32 and the
chunkwisebackend (maximum absolute difference 0.0). - Direct loading from the public Hugging Face repository produced the same 4-token greedy continuation as loading the locally downloaded weights (
' Paris, which is'for the test prompt). - This is a small pretrained research model. It can generate inaccurate, repetitive, biased, unsafe, or nonsensical text; it is not suited to safety-critical or consequential use.
- No instruction following, factual reliability, or downstream task performance has been established through public benchmarks. The 2,048-token configuration limit should not be confused with demonstrated performance at that length.
- The project code implements the architecture, but the optimized CUDA KDA path uses the upstream Flash-Linear-Attention implementation; it is not a wholly original Triton kernel.
References and attribution
- Source code and tests: adriaflores/kimi-k3-from-scratch
- Architecture, training and results: kimi-k3.adriaflores.com
- Kimi K3 paper: Kimi K3: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Pretraining corpus: FineWeb-Edu
- Tokenizer: SmolLM2-135M
The implementation is licensed under Apache License 2.0; this release declares Apache 2.0 for the model weights as well. Third-party code, data, and tokenizer assets remain subject to their respective upstream terms.