Huihui-Qwen3.8-Flash-Next-abliterated-exl3-4bit-hq_h6_ng6

This is a 4.05 bpw EXL3 HQ quantization of a corrected Huihui Qwen3.8 Flash Next checkpoint. It includes MTP, vision and the n-gram embedding table.

I rebuilt the BF16 source with ablation strength1.0 before quantization. These weights contain the correction and need no runtime ablation hook. The public repo name omits strength1.0, but the correction is part of the model you download.

Source correction

The source is huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated at revision 298f94632b784e26a7fe576114f82066689d5baa. I applied the following projection to 144 residual-writing tensors across its 48 trunk layers, using the fitted unit direction r:

W_new = W_Huihui - r (r^T W_Huihui)

The calculation runs in FP32 and writes back BF16. BF16 rounding leaves a small residual component. MTP, vision and n-gram source tensors were unchanged. strength1-patch-provenance.json includes the direction SHA256, affected tensor names and measured residual after rounding.

Quantization

EXL 1.5.3 produced this checkpoint with fresh calibration of 250 rows of 2048 tokens. The n-gram table reuses my unchanged K6 table from the corrected Q4 conversion.

The converter reported 4.05 bpw. The converter output totaled 107,463,164,330 bytes, including the 39,040,214,168-byte n-gram table. With ngram_ram: true, that table loads into system RAM. Download size is therefore larger than GPU weight memory. SHA256SUMS covers the packaged files.

Validation

On four RTX 3090s, I manually reviewed reasoning, a tool call followed by a real tool reply, and vision answers with MTP off and MTP2. Both modes also completed two turns of an OMP coding task at xhigh. The agent edited the code, ran the four original tests after each turn and produced coherent final answers. Independent checks covered the implementation and CLI.

An additional Range task passed 31 agent tests with MTP off and 38 with MTP2. The same 18 original tests passed independently in both modes. The MTP-off agent created and removed /tmp/orig_check despite a project-only instruction. Functional tests passed. That containment instruction was violated. These results cover the tested tasks only.

The speed run used eight prompt lengths from 4096 to 261632 tokens, C1 and C2, and two repeats in each MTP mode. Each request generated 512 actual tokens with zero reported cached prefix tokens. At the longest point, every request reached 261632 + 512 = 262144 total tokens. The two C2 requests had independent prompts and overlapping decode.

Full-context retrieval returned all three exact secrets in each C1/C2 request. Those answers used 57 or 58 output tokens. Full-context vision also reached 262144 total tokens per request in two C1/C2 repeats, with the shapes, colors and positions checked manually. The same image was reused, so its image-embedding cache may have been warm. The 512-token continuations were reviewed as excerpts. They do not establish complete long essays or factual accuracy of every claim.

Selected measurements on 4× RTX 3090 at 300 W per GPU. Decode is the median tokens/s per request. TTFT is seconds until the first token. C2 means two concurrent requests.

At 261632 prompt tokens in C1, median end-to-end time including prefill was 165.23 s with MTP off and 169.71 s with MTP2. Faster decode did not reduce total latency for that test.

The speed curve uses greedy nonthinking generation. Thinking quality tests use the sampler below. C2 decode rates are per-request rates, and changing MTP also changes the automatic layer placement. These measurements describe the tested serving configurations.

Sampling and serving

The tested thinking sampler uses these OpenAI-compatible request fields:

{
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95,
  "min_p": 0.0,
  "presence_penalty": 0.0,
  "repetition_penalty": 1.0,
  "reasoning_effort": "medium",
  "chat_template_kwargs": {"enable_thinking": true}
}

I checked low, medium and xhigh on the actual OMP requests. Serve with a compatible EXL3 runtime and the included chat template, vision enabled, ngram_ram: true and output_chunking: false.

The tested server used a 262144-token sequence limit, a 540672-token FP16 KV-cache, a maximum batch size of two and automatic layer placement across the four GPUs. CPU MoE expert offload was disabled. Sampled mode-wide GPU memory peaked at 82620 MiB with MTP off and 85670 MiB with MTP2, summed across all four GPUs.

The recorded runtime was EXL 1.5.3+cu128.torch2.8.0 at commit d3739fd393337b1ff4d6c2a342b12f0c87a9592f, TabbyAPI at be74bf0a00bcb3a518e6feb7606f150c189be637, Torch 2.8.0+cu128 and Transformers 5.13.1.

Other sizes and license

The separate Q3, Q4 and Q5 repos use the same corrected BF16 source.

These weights retain the Qwen Community License 1.0 shipped with the source checkpoint. Read that file for its terms.