LLaDA2.2-mini — Jetson Orin Nano Super 8GB (Packed Mixed 2/3/4-Bit Pure-CUDA Build)

Mixed 2/3/4-bit Hadamard-rotated GPTQ + 8-bit non-expert pure-CUDA build of inclusionAI/LLaDA2.2-mini (16B MoE diffusion LLM, 1.4B active) serving 32K context at ~150–180 tok/s on an 8 GB NVIDIA Jetson Orin Nano Super.

---

inclusionAI/LLaDA2.2-mini (16B MoE diffusion LLM, 1.4B active) quantized and packed for the pure-CUDA engine patrickbdevaney/llada2-orin-cuda, which serves it OpenAI-compatibly on a Jetson Orin Nano Super 8GB with 32K context.

These files are not a transformers checkpoint: they are the engine's tile format and need that repo to run.

---

✨ Highlights

• 16B MoE Diffusion LLM in 8 GB Unified Memory with 32K Context: Packs inclusionAI/LLaDA2.2-mini (16B total, 1.4B active) + a 32K-token int4 KV cache into the 8 GB unified memory budget of a Jetson Orin Nano Super.
• ~150–180 tok/s Parallel Diffusion Decoding: Runs a 32-token denoising step in ~25 ms at 1K context on pinned Orin Nano Super clocks, delivering ~180 tok/s on math/code/tool-call tasks at the server's default decode settings and ~150 tok/s at the model card's settings.
• Hadamard-Rotated Mixed 2/3/4-Bit GPTQ Experts + 8-Bit Non-Expert Path: Routed experts are quantized with group-128 GPTQ in a Hadamard-rotated basis (experts_L*.bin, per-expert bit widths in experts.json), while attention, shared experts, layer-0 dense MLP (nonexpert_L*.bin), and lm_head.bin (with high/low nibble split planes in lm_head_split.bin for screened readout) are preserved at 8-bit and host-mapped embed.bin at FP16.
• 113–116 / 116 Scored Task Suite: Achieves 113–116 / 116 across the engine's 96 generated math problems, 12 unit-tested code tasks, and 8 tool calls.

---

📋 Model Summary

---

What is in here

• experts_L*.bin: routed experts, GPTQ (group 128) in a Hadamard-rotated basis, per-expert bit widths chosen against a 32K-context memory budget (mostly 2-bit, some 3- and 4-bit; ./experts.json lists them).
• nonexpert_L*.bin: attention, shared expert and the dense layer-0 MLP, 8-bit GPTQ.
• ./lm_head.bin (8-bit) and ./lm_head_split.bin (the same codes split into high/low nibble planes, used by the engine's screened readout); ./embed.bin (fp16 embeddings, memory-mapped on the host); ./small.bin (norms, router, rotation signs); ./engine.idx (the index the engine loads); ./model.json (model metadata).
• Tokenizer (./tokenizer.json, ./tokenization_llada2.py), ./chat_template.jinja, and ./tool_declaration_ts.py from the base model, unchanged, so this directory is also the server's --model directory.

KV cache at runtime: int4 (keys per channel per 32-token block, values per token).

---

Use

git clone https://github.com/patrickbdevaney/llada2-orin-cuda && cd llada2-orin-cuda/engine && make
huggingface-cli download patrickbdevaney/LLaDA2.2-mini-orin-nano --local-dir ~/models/llada2mini-orin
cd ../server && python3 server.py --model ~/models/llada2mini-orin --engine ~/models/llada2mini-orin

On the Orin Nano Super with pinned clocks: ~25 ms per 32-token denoising step at 1K context, ~180 tok/s on math/code/tool-call tasks at the server's default decode settings, ~150 tok/s at the model card's.

---

Quality

Measured against the BF16 model on held-out text, the quantization costs mostly in the routed experts (see QUANT_PLAN.md in the code repo). On the engine's scored test (96 generated math problems, 12 code tasks checked against tests, 8 tool calls) it scores 113-116 / 116. The misses have correct arithmetic and a repeated final digit; occasional repetition artifacts like this are the main visible cost of the 2-bit experts.

---

🔎 Provenance & Verification Ledger

Every claim, bit-width allocation, and kernel benchmark on this card can be audited directly against the files in this repository and the companion pure-CUDA engine repository:

---

License

Apache-2.0, following the base model (inclusionAI/LLaDA2.2-mini). Credit for the base model: inclusionAI, LLaDA2.2. Engine & quantization: patrickbdevaney/llada2-orin-cuda.