Sign up

com-kotobalabs/Nex-N2.5-mini-Uncensored-W4A16-AutoRound

🤗 Hugging Face sourcetext-generationapache-2.06B params137 GBsafetensors✓ Checksums witnessedupdated today
No torrent yet →

Nex-N2.5-mini-Uncensored-W4A16-AutoRound

W4A16 (INT4 weights, group size 128, symmetric, BF16 activations) quantization of orcarouter/Nex-N2.5-mini-Uncensored (the abliterated Nex-N2.5-mini, a Qwen3.5-MoE 35B-A3B vision-language model) in auto_gptq format, loadable by vLLM (Marlin / XPU) and SGLang.

Why this exists

orcarouter publishes the Uncensored weights as BF16 (65 GiB), FP8 (34 GiB, Hopper+), NVFP4 (22 GiB, Blackwell only), MLX (Apple) and GGUF (llama.cpp). None of those loads in vLLM on a 32 GiB non-Blackwell card. [quant-mind/Nex-N2.5-mini-W4A16-AutoRound] (https://huggingface.co/quant-mind/Nex-N2.5-mini-W4A16-AutoRound) filled that gap for the STOCK model; this repository applies the same published recipe to the Uncensored weights, so the two differ in base weights only. Built for the murakumo fleet's Intel Arc Pro B70 (32 GiB) head, where vLLM XPU + XPU graph measured ~1,128 tok/s aggregate on the stock W4A16 checkpoint vs ~137 tok/s for llama.cpp IQ4_XS on the same card (root ADR-2609181615, com-junkawasaki/root).

Recipe (identical to quant-mind's config.json)

  • Intel AutoRound 0.15.1, --scheme W4A16 --group_size 128 (sym), --iters 200, --nsamples 128 --seqlen 2048, calibration NeelNanda/pile-10k, --low_gpu_mem_usage
  • Kept in BF16: mlp.gate (router), mlp.shared_expert_gate, mtp, lm_head, embeddings, the visual tower
  • Quantized: linear_attn.*_proj, self_attn.{q,k,v,o}_proj, all 256 mlp.experts.*, mlp.shared_expert.*
  • Export format: auto_gptq
  • Built on Modal (H100) by the job tools/modal-quant/nex_n25_mini_uncensored_w4a16.py in kotoba-lang/murakumo.

Use

vllm serve com-kotobalabs/Nex-N2.5-mini-Uncensored-W4A16-AutoRound     --reasoning-parser qwen3 --max-model-len 65536 --gpu-memory-utilization 0.90

The model thinks before answering; without --reasoning-parser qwen3 the reasoning leaks into content.

Caveats

  • Abliterated / uncensored derivative: it will not refuse. Deploy behind your own policy layer; the murakumo gateway runs it only behind governed organisms.
  • Quality vs the BF16 source is not evaluated here beyond a coherence probe; the quantization error profile is the one AutoRound publishes for this recipe.
  • License follows the base: Apache-2.0 (nex-agi/Nex-N2.5-mini) as relicensed by orcarouter.