Huihui-Qwen3.8-Flash-Next-abliterated-exl3-5bit-hq_h6_ng6

This is a 5.05 bpw EXL3 HQ quantization of a corrected Huihui Qwen3.8 Flash Next checkpoint. It includes MTP, vision and the n-gram embedding table.

I rebuilt the BF16 source with ablation strength1.0 before quantization. These weights contain the correction and need no runtime ablation hook. The public repo name omits strength1.0, but the correction is part of the model you download.

Source correction

The source is huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated at revision 298f94632b784e26a7fe576114f82066689d5baa. I applied the following projection to 144 residual-writing tensors across its 48 trunk layers, using the fitted unit direction r:

W_new = W_Huihui - r (r^T W_Huihui)

The calculation runs in FP32 and writes back BF16. BF16 rounding leaves a small residual component. MTP, vision and n-gram source tensors were unchanged. strength1-patch-provenance.json includes the direction SHA256, affected tensor names and measured residual after rounding.

Quantization

EXL 1.5.3 produced this checkpoint with fresh calibration of 250 rows of 2048 tokens. The n-gram table reuses my unchanged K6 table from the corrected Q4 conversion.

The converter reported 5.05 bpw. The converter output totaled 122,928,921,831 bytes, including the 39,040,214,168-byte n-gram table. With ngram_ram: true, that table loads into system RAM. Download size is therefore larger than GPU weight memory. SHA256SUMS covers the packaged files.

Validation

This Q5 release passed conversion structure checks and has a saved SHA256 manifest. Initial MTP-off tests with Q8 KV-cache passed short reasoning, an actual tool-call roundtrip and vision answers. The OMP coding agent completed 2 turns at xhigh and ran the 4 original tests after each edit. Independent implementation and CLI checks also passed.

The agent made errors in its final explanation of the old average and Python exception type. The actual code and tests passed. It also wrote a /tmp file despite a project-only instruction. These results cover the sampled functional behavior only.

Speed on 4x RTX 3090

I measured Q5 with MTP off and MTP2 on 4x RTX 3090 at 300 W per GPU, using EXL 1.5.3 and TabbyAPI. Both configurations use a Q4 main KV-cache of 270336 tokens, batch size 1 and the n-gram table in system RAM. MTP2 also uses Q4 for its draft KV-cache. CPU MoE expert offload was disabled.

Each request generated 512 tokens with temperature 0 and top_k 1. These are nonthinking speed measurements with 1 request at a time. I used matching prompt texts across both modes, 0 cached prompt tokens and 2 repeats per point. The table reports arithmetic means.

At the last point, each request used 261632 prompt tokens plus 512 output tokens, reaching the 262144-token limit. EXL autosplit changes the layer placement when MTP is enabled, so these numbers compare the 2 serving configurations. The measured groups, TTFT and end-to-end timings are in benchmark-speed.json.

Sampling and serving

The tested thinking sampler uses these OpenAI-compatible request fields:

{
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95,
  "min_p": 0.0,
  "presence_penalty": 0.0,
  "repetition_penalty": 1.0,
  "reasoning_effort": "medium",
  "chat_template_kwargs": {"enable_thinking": true}
}

I checked low, medium and xhigh on the actual OMP requests. Serve with a compatible EXL3 runtime and the included chat template, vision enabled, ngram_ram: true and output_chunking: false.

The earlier MTP-off tests used Q8 KV-cache. The speed measurements above use Q4 KV-cache. The included n-gram table loads into system RAM with ngram_ram: true.

Other sizes and license

The separate Q3, Q4 and Q5 repos use the same corrected BF16 source.

These weights retain the Qwen Community License 1.0 shipped with the source checkpoint. Read that file for its terms.