Nanbeige 4.2 3B — CMF q4t (2.4 GB, one file, no Python)

Nanbeige/Nanbeige4.2-3B converted to the CMF format: a single mmap-served file executed by a pure-Rust runtime.

Nanbeige 4.2 is a Looped Transformer: 22 physical layers applied twice (44 virtual layers) with a re-norm at each loop boundary — the engine executes the loop natively — and a reasoning model that thinks before it answers. On an RTX A5000 with CUDA it decodes at 106 tok/s and reads a 4096-token prompt in 1.4 s.

Speed

RTX A5000 (24 GB), CUDA

cortiq 0.8.96, cortiq bench, CMF_GPU=cuda on a build with the CUDA backend (cargo install cortiq-cli --features cuda, the NVIDIA driver and the CUDA 12.x libraries):

Vulkan on the same card: decode 63.4–63.6 tok/s, the 4096-token prompt 254 tok/s (first token 16.1 s).

Apple M4

Measured on a fanless MacBook Air (Apple M4) — 10-core CPU (4 performance + 6 efficiency), 10-core GPU, 24 GB unified memory, macOS 27. No external cooling, no overclock, nothing plugged in.

Method: cortiq bench --core (greedy, no repetition penalty — the llama-bench contract), median of three runs on an idle machine. The GPU column is native Metal, enabled with CMF_GPU=1. Needs cortiq ≥ 0.5.35 — earlier releases decode this file at ~9 tok/s and take 40 s to ingest a 512-token prompt.

Why a "3B" runs at these rates and not faster. The loop means every token pays for 44 layer passes, ~6.3 G multiply-accumulates — the arithmetic of a 6B dense model. The 166k vocabulary with untied embeddings adds another 1.0B parameters, so the file really holds 4.17B. At q4t that is ~3.9 GB of weights read per token, and an M4 moves about 120 GB/s, which puts a hard ceiling near 31 tok/s on this machine no matter how good the kernels are. 22.4 tok/s is ~72% of that ceiling. A laptop with more memory bandwidth (M4 Pro/Max, or a discrete GPU through the Vulkan backend) scales roughly with it.

Run it

cargo install cortiq-cli          # pure Rust, no Python; needs >= 0.5.35
hf download infosave/Nanbeige4.2-3Bcmf nanbeige42-3b-q4t.cmf --local-dir .

CMF_GPU=1 cortiq run nanbeige42-3b-q4t.cmf \
  --prompt "What is the capital of France?" --max-tokens 300

CMF_GPU=1 cortiq serve nanbeige42-3b-q4t.cmf   # OpenAI-compatible API

CMF_GPU=1 selects native Metal on Apple Silicon; on Linux/Windows the same flag picks the Vulkan/DX12 backend. Without it the runtime stays on the CPU, which is the second column above.

Reasoning models deliberate first — give them a few hundred --max-tokens to reach the final answer.

Reproduce the conversion

cortiq convert --model Nanbeige/Nanbeige4.2-3B --quant q4t --output nanbeige42-3b-q4t.cmf

The converter streams the checkpoint shard by shard, so the free-disk requirement is one shard plus the output.

Fidelity

The looped forward is checked against a float32 PyTorch reference built from the upstream modeling code: same argmax and same top-5 ordering on the same prompt, with the q4t logits within ~2 of the reference's. The model's own quirks survive the quantization intact — including a reasoning opener it is very sure about.

License

Apache-2.0, inherited from the base model. Weights © Nanbeige LLM Lab; this repackaging only changes the storage format.

Run it on a phone

CMF Mobile is a Flutter app built on this runtime: install it, point it at this 2.4 GB .cmf and chat on-device with streaming tokens — or flip on its server mode and the phone becomes an OpenAI-compatible endpoint for your network. The engine, converter and format spec live at https://github.com/infosave2007/cmf.