Nex-N2.5-mini-Uncensored-W4A16-AutoRound

W4A16 (INT4 weights, group size 128, symmetric, BF16 activations) quantization of orcarouter/Nex-N2.5-mini-Uncensored (the abliterated Nex-N2.5-mini, a Qwen3.5-MoE 35B-A3B vision-language model) in auto_gptq format, loadable by vLLM (Marlin / XPU) and SGLang.

Why this exists

orcarouter publishes the Uncensored weights as BF16 (65 GiB), FP8 (34 GiB, Hopper+), NVFP4 (22 GiB, Blackwell only), MLX (Apple) and GGUF (llama.cpp). None of those loads in vLLM on a 32 GiB non-Blackwell card. [quant-mind/Nex-N2.5-mini-W4A16-AutoRound] (https://huggingface.co/quant-mind/Nex-N2.5-mini-W4A16-AutoRound) filled that gap for the STOCK model; this repository applies the same published recipe to the Uncensored weights, so the two differ in base weights only. Built for the murakumo fleet's Intel Arc Pro B70 (32 GiB) head, where vLLM XPU + XPU graph measured ~1,128 tok/s aggregate on the stock W4A16 checkpoint vs ~137 tok/s for llama.cpp IQ4_XS on the same card (root ADR-2609181615, com-junkawasaki/root).

Recipe (identical to quant-mind's config.json)

• Intel AutoRound 0.15.1, --scheme W4A16 --group_size 128 (sym), --iters 200, --nsamples 128 --seqlen 2048, calibration NeelNanda/pile-10k, --low_gpu_mem_usage
• Kept in BF16: mlp.gate (router), mlp.shared_expert_gate, mtp, lm_head, embeddings, the visual tower
• Quantized: linear_attn.*_proj, self_attn.{q,k,v,o}_proj, all 256 mlp.experts.*, mlp.shared_expert.*
• Export format: auto_gptq
• Built on Modal (H100) by the job tools/modal-quant/nex_n25_mini_uncensored_w4a16.py in kotoba-lang/murakumo.

Use

vllm serve com-kotobalabs/Nex-N2.5-mini-Uncensored-W4A16-AutoRound     --reasoning-parser qwen3 --max-model-len 65536 --gpu-memory-utilization 0.90

The model thinks before answering; without --reasoning-parser qwen3 the reasoning leaks into content.

Caveats

• Abliterated / uncensored derivative: it will not refuse. Deploy behind your own policy layer; the murakumo gateway runs it only behind governed organisms.
• Quality vs the BF16 source is not evaluated here beyond a coherence probe; the quantization error profile is the one AutoRound publishes for this recipe.
• License follows the base: Apache-2.0 (nex-agi/Nex-N2.5-mini) as relicensed by orcarouter.

External inference evaluation (2026-09-24)

The W4A16 AutoGPTQ checkpoint loaded with vLLM 0.29.0 XPU on an Intel Arc Pro B70, using the XPU WNA16 kernel and 19.18 GiB for model weights. A 10-token smoke response completed, but a following arithmetic probe caused Level Zero DEVICE_LOST; this is not a stable or qualified throughput result. The 18.14 tok/s long-document result reported for a Radeon 680M belongs to a separate Q3_K_M APEX-Mini GGUF derived from the same uncensored base model, not this W4A16 checkpoint. Do not treat it as this artifact's speed. Full evidence and compatibility limits: https://github.com/kotoba-lang/inference/blob/main/verify/evidence/model-inference-speed-analysis-20260924.md.

Publisher

Published by Mithril (formerly Kotoba Cloud), operated by Kotoba Labs Inc. The Hugging Face handle com-kotobalabs is unchanged. Support: support.mithril.fund.