Huihui-Qwen3.8-Flash-Next-abliterated-exl3-3bit-hq_h6_ng6

This is a 3.05 bpw EXL3 HQ quantization of a corrected Huihui Qwen3.8 Flash Next checkpoint. It includes MTP, vision and the n-gram embedding table.

I rebuilt the BF16 source with ablation strength1.0 before quantization. These weights contain the correction and need no runtime ablation hook. The public repo name omits strength1.0, but the correction is part of the model you download.

Source correction

The source is huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated at revision 298f94632b784e26a7fe576114f82066689d5baa. I applied the following projection to 144 residual-writing tensors across its 48 trunk layers, using the fitted unit direction r:

W_new = W_Huihui - r (r^T W_Huihui)

The calculation runs in FP32 and writes back BF16. BF16 rounding leaves a small residual component. MTP, vision and n-gram source tensors were unchanged. strength1-patch-provenance.json includes the direction SHA256, affected tensor names and measured residual after rounding.

Quantization

EXL 1.5.3 produced this checkpoint with fresh calibration of 250 rows of 2048 tokens. The n-gram table reuses my unchanged K6 table from the corrected Q4 conversion.

The converter reported 3.05 bpw. The converter output totaled 91,997,490,918 bytes, including the 39,040,214,168-byte n-gram table. With ngram_ram: true, that table loads into system RAM. Download size is therefore larger than GPU weight memory. SHA256SUMS covers the packaged files.

Validation

I reviewed short reasoning, an actual tool-call roundtrip and vision answers with MTP off and MTP2. Both modes completed 2 turns of an OMP coding task at xhigh, with 4 original tests run by the agent after each turn. Independent checks covered the code and CLI.

The MTP-off final gave an incorrect old average in its explanation. The corrected implementation and tests were right. In MTP2, the agent redirected error-path output to files in /tmp despite a project-only instruction. These samples passed the functional checks but do not establish error-free explanations or strict file containment.

The short tests used a 262144-token server limit, a 540672-token FP16 KV-cache and batch size 2.

Speed on 4x RTX 3090

I measured Q3 with MTP2 on 4x RTX 3090 at 300 W per GPU, using EXL 1.5.3 and TabbyAPI. The FP16 KV-cache holds 540672 tokens, and the n-gram table loads into system RAM. CPU MoE expert offload was disabled.

Each request generated 512 tokens with temperature 0 and top_k 1. These are nonthinking speed measurements. I ran each point 2 times with 0 cached prompt tokens. The table reports arithmetic means across those repeats.

C1 is 1 request. C2 is 2 requests with different prompt prefixes and measured overlap during generation. The C2 sum adds the server-reported decode rates of both requests; it excludes prefill and is not end-to-end throughput.

At the last point, every request used 261632 prompt tokens plus 512 output tokens, reaching the 262144-token limit. I reviewed all 48 saved output excerpts and saw no sustained loops. The measured groups, TTFT and end-to-end timings are in benchmark-mtp2.json.

Sampling and serving

The tested thinking sampler uses these OpenAI-compatible request fields:

{
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95,
  "min_p": 0.0,
  "presence_penalty": 0.0,
  "repetition_penalty": 1.0,
  "reasoning_effort": "medium",
  "chat_template_kwargs": {"enable_thinking": true}
}

I checked low, medium and xhigh on the actual OMP requests. Serve with a compatible EXL3 runtime and the included chat template, vision enabled, ngram_ram: true and output_chunking: false.

I used EXL 1.5.3 with TabbyAPI on 4x RTX 3090 at 300 W per GPU. The n-gram table loaded into system RAM, and CPU MoE expert offload was disabled.

Other sizes and license

The separate Q3, Q4 and Q5 repos use the same corrected BF16 source.

These weights retain the Qwen Community License 1.0 shipped with the source checkpoint. Read that file for its terms.