FWKV

FWKV (Feed-forward Weighted Key Value, or Floored Weighted Key Value) is a novel efficient language model architecture proposed on May 13, 2026 by Me.

Design Philosophy

FWKV is built on two simple ideas:

• FFNN as the primary compute block — fast, parallelizable, hardware-friendly
• floor(W·K·V) as the sole context mechanism — a multiplicative recurrent state with a learned decay W floored above zero, so the model can never fully forget

Sample

Inference speed (CUDA -- T4)

Training

Factorised tied weights, chunked CE, bf16 training. Best val perplexity: 143.62