diffusiongemma-26B-A4B-it — fni8 (int8/W4A8 dp4a, Volta sm_70)

⚠️ EXPERIMENTAL — do not rely on this checkpoint.

It was converted with a GeGLU-run-as-SiLU converter bug (now fixed), so its MLP activations are wrong. A corrected reconversion is pending. It also requires the engine's DiffusionDecodeStrategy before it can run at all.

DiffusionGemma (diffusion LLM, MoE), quantized from google/diffusiongemma-26B-A4B-it. Repackaged to the .fni8 resident format (~50 GB (int8)) for the fni8 W8A8/W4A8 DP4A kernels on NVIDIA Volta (sm_70) — Tesla V100 / CMP 100-210.

Status

See the experimental warning above. Not validated end-to-end.

Format

• Weights: int8 per-row (W8A8), fp32 scales, resident dp4a VRAM layout.
• Why dp4a: sm_70 has no int8 tensor cores, so the matmul contraction runs on the __dp4a CUDA-core intrinsic. On the CMP 100-210 fleet (whose fp16 tensor cores are firmware-limited) dp4a is the fast path, not a compromise.

How to run

fni8-serve is the LLM runtime (load_fni8_state_dict(<file>) into an LLMEngine; the architecture is read from the file). ComfyUI-fni8 is for diffusion DiTs only and does not load this model.

Limitations

• Quantization is lossy: int8 (and especially int4) outputs differ from the fp16/bf16 parent, and the difference varies by task.
• Capabilities, biases, and risks of the parent model carry over — see the parent card.
• This is a derivative quantization, not a relicense; the parent model's license and acceptable uses apply.

---

Part of the fni8 stack: kernels · LLM serving · ComfyUI DiTs.