Qwen3.8 27B

NVFP4 · embedded MTP · Preserved candidate · quality ungraded

BennyDaBall's native NVFP4 GGUF with the MTP head retained, derived from JonathanColetti's modified Qwen3.8 27B checkpoint.

Benchmark results · Files · Download · Runtime · Provenance · Full collection

Join Infinity on Discord

Get model support, request models and quants, discuss benchmarks, and follow new releases and model improvements.

Join the Infinity community

All model tests, per-question scores, experiment history and scoring methods

Qwen effort study — separate full-404 cohort: medium 82.87/100 versus xhigh 84.41/100; paired difference +1.54 points (95% interval [-1.71, +4.82]). 404 collected questions, 403 valid questions graded per effort; one invalid reference annotation excluded symmetrically. The historical 202-question cross-model comparison below is unchanged. Study, timing and blind subjective results.

Preserved and documented by isilicon / Mango. The original model, modification and quantization remain the upstream creators' work. This repository adds attributed preservation and exact-configuration evaluation, with the selected source bytes unchanged.

At a glance

GB is decimal and GiB is binary. Payload size is disk/storage size, not a claim that the model fits in the same amount of VRAM. Runtime, KV cache, working memory and companions add their own requirements.

Mango evaluation

No standalone Mango quality score has been published for these exact files. This candidate is preserved, but its completed matched grade is not part of this card release.

Related configurations measured separately

These results describe the linked configurations, not this artifact. A change of quant, modification lineage, draft, engine or tokenizer can change the result.

A completed score requires a native preflight, matching frozen prompts, 202 suite answers, 12 agent cases, archive verification and the same reference graders. The benchmark methodology describes the protocol; publisher-side results are retained separately below.

Runtime and compatibility

The native FP4 path requires Blackwell support, compatible NVFP4 CUDA kernels and an architecture-aware runtime. MTP additionally requires draft-mtp support. The source vision projector is not part of this selected mirror.

Mango's exact-file quant/speed comparison is pending; no MLX4, Q8 or BF16 score transfers here.

The local evaluation covers text/tool tasks. It does not establish the full advertised context ceiling, image/audio quality or every optional speculative-decoding path. Publisher capabilities and Mango-verified behavior are separate scopes.

Companion artifacts

No separate companion mirror is declared in this selected manifest. This does not imply that every optional upstream projector, draft head or multimodal path is included; consult the exact file list.

Files and integrity

Full paths, byte sizes and complete SHA-256/Git blob identities are in SOURCE_MANIFEST.json. Available original README/license/notice files are also preserved. MIRROR_STATUS.json records the earlier source-file verification; MODEL_CARD_STATUS.json records this documentation refresh separately.

Download and use

The commands pin the already-verified mirror weight revision. Documentation can advance on main without changing that preserved weight snapshot. Keep original subfolders and all shards.

python -m pip install -U huggingface_hub

hf download isilicon/mango-qwen38-27b-nvfp4mtp \
  Qwen3.8-27B-Uncensored-NVFP4-MTP.gguf \
  --revision 69118ce138e7a77b1011724cec2699fb10d8eec2 \
  --local-dir ./weights/qwen38-27b-nvfp4mtp

Load example and setup scope

llama-server \
  --model ./weights/qwen38-27b-nvfp4mtp/Qwen3.8-27B-Uncensored-NVFP4-MTP.gguf \
  --host 127.0.0.1 --port 8080 --ctx-size 32768 \
  --jinja --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0

This is a load-command pattern, not a validated Mango inference or performance result for this candidate. Check architecture support before loading; enable projector/MTP options only after their own preflight.

Provenance and license

The publisher describes native FP4 transformer weights with selected tensors, including the draft head, retained at higher precision. Its RTX 5090 speed table is a publisher measurement, separate from Mango's hardware and protocol.

Original notices, base-model conditions and the publisher's stated limitations still apply. Downloadable preservation does not add new license rights. Source-described refusal modification and publisher-side benchmark claims have not been independently re-measured by this Mango task cohort.

Original publisher documentation

The unchanged upstream README and pinned source card retain the original usage guidance, technical discussion, attribution, caveats and citations.

Read the complete original publisher card

Source reference. The following is the original publisher's documentation. Its other quants, full-model files, context claims and benchmark tables can exceed the selection in this mirror. Relative asset/file URLs are qualified to the pinned original source; publisher measurements remain separate from Mango measurements.

Qwen3.8-27B-Uncensored NVFP4 (MTP)

Follow me on X @BennyDaBall_OG !

A native NVFP4 GGUF of JonathanColetti/Qwen3.8-27B-Uncensored, the Heretic-abliterated build of Qwen3.8-27B. The whole transformer backbone is quantized to 4-bit NVFP4 for Blackwell, and the MTP (multi-token prediction) speculative head is kept intact for fast decoding. No retraining, no distillation, just a clean quant.

---

What is this?

• 27B, uncensored, NVFP4. Every attention, Gated DeltaNet, and MLP weight across all 64 layers is quantized to native NVFP4 (GGML type 40). The lm_head, token embeddings, and the MTP draft head stay in BF16.
• MTP head retained. The model ships as 65 blocks (blk.64 is the MTP draft head), so llama.cpp can self-speculate with --spec-type draft-mtp for a large decode speedup, no separate draft model needed.
• Vision included. The repo ships the BF16 vision projector (mmproj-BF16.gguf), so the model sees images and video frames when loaded with --mmproj. The abliteration only modified text-model tensors, so the vision tower is stock Qwen3.8 quality.
• 262,144 native context.

Use this file when you want a fast, uncensored 27B on a Blackwell GPU (RTX 5090, 5080, RTX PRO) with native FP4 density and built-in speculative decoding.

---

The files

sha256 (model):  db17acbca53da5a7b0e861175b198cc6fd467865e99d1ad53d9dca584257a1a1
sha256 (mmproj): 5ac423f8a29059dc24e51bc6a43e9380dcd57a9347f28b62591e0b3f60b7081c

The corrected chat template is embedded in the model file. Text-only use works without the mmproj; add it when you want image input (~1 GiB extra VRAM).

---

Requirements

• A Blackwell (sm_120) GPU for the native FP4 path.
• A recent llama.cpp with NVFP4 CUDA kernels and the qwen35 architecture, or LM Studio 2.29.1+ (runtime llama.cpp-nvidia-cuda12 2.29.1 loads it and runs coherently).
• For the MTP speedup, a build with the draft-mtp speculative path.

---

Quick Start

llama.cpp / llama-server

llama-server \
  --model Qwen3.8-27B-Uncensored-NVFP4-MTP.gguf \
  --mmproj mmproj-BF16.gguf \
  --ctx-size 229376 \
  --flash-attn on \
  -ctk q8_0 -ctv q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-split 0.2 \
  --temp 1.0 --top-p 0.95 --top-k 20

Measured on an RTX 5090 (131k context, q8 KV, greedy, 512-token generations, mean of 2 runs): 105.8 / 136.3 / 123.0 tok/s on prose / code / 12k-deep context versus 75.2 / 74.0 / 72.2 with speculation off, a 1.4-1.8x lift. MTP acceptance is hardware and prompt dependent. Sweep --spec-draft-n-max from 1 to 6 and keep whatever is fastest on your box. Drop the --mmproj line (or pass --no-mmproj) for a text-only server with a little more VRAM headroom.

LM Studio

Load the file and chat. Two things to know:

• Thinking is on by default. Give it a generous max-tokens budget or the whole budget can be spent inside the hidden reasoning block and the visible answer comes back empty. The old Qwen3 /no_think trick does not work on this template (LM Studio returns the whole budget as reasoning_content); on llama-server, disable thinking with chat_template_kwargs: {"enable_thinking": false}.
• MTP speculation works here too, but tune it. LM Studio (runtime 2.29.1+) detects the embedded MTP head and turns speculation on automatically, but at the default settings it drafts too short to gain any speed. Set max draft tokens 3 and continue probability 0.2, in the UI speculative decoding settings or at load time with lms load ... --speculative-draft-mtp --speculative-draft-max-tokens 3 --speculative-draft-min-continue-probability 0.2. Measured on an RTX 5090 (131k context, q8 KV, greedy, 512-token generations, prose / code / 12k-deep): speculation off 71.1 / 71.2 / 69.1 tok/s, tuned 79.9 / 88.9 / 89.3 tok/s. Short drafts win here; max tokens 2-3 with probability 0.1-0.2 is the sweet spot, and max tokens above 4 or probability 0.3+ is slower than speculation off. The llama-server draft-mtp path is faster still on the same file.

---

Performance (RTX 5090, single-stream)

Measured with the llama.cpp draft-mtp path, q8_0 KV cache, flash attention on. Single-run figures, not a formal benchmark.

MTP self-speculation reaches high draft acceptance on structured output (counting, code, repetitive passages) and lifts decode well above the no-speculation baseline. Acceptance and speedup drop on high-entropy creative prose, as expected.

---

What is NVFP4?

NVFP4 is NVIDIA's 4-bit floating-point weight format for Blackwell tensor cores: 16-element blocks, each with an FP8 (E4M3) block scale plus a global tensor scale. It keeps more of the weight distribution than integer 4-bit and runs on the FP4 tensor cores, so the whole backbone stays dense at 4-bit on a Blackwell card.

---

How it was made

1. Started from the JonathanColetti/Qwen3.8-27B-Uncensored BF16 checkpoint.
2. Converted to a BF16 GGUF parent (MTP head preserved as blk.64).
3. Quantized the transformer backbone to NVFP4 with an importance matrix, keeping lm_head, embeddings, and the MTP head in BF16.
4. The vision projector is the base model's own BF16 projector (the abliteration never touched the vision tower), verified working against this quant (image in, correct description out).

No weights were trained or fine-tuned. The abliteration and all behavior come from the base model; quantization is a transformation only.

---

Notes

• Uncensored / abliterated. Refusal behavior is inherited from the base model (Heretic abliteration), not from this quant. You are responsible for how you use it.
• Blackwell only for the native FP4 path.

---

Acknowledgements

• Alibaba / Qwen for Qwen3.8-27B.
• JonathanColetti for the uncensored base, Qwen3.8-27B-Uncensored.
• The maintainers behind llama.cpp and LM Studio.

Apache-2.0, same as every upstream artifact. "Qwen" is a trademark of Alibaba, used only to identify the upstream model; this repo is not affiliated with or endorsed by Alibaba.

Quantized locally by BennyDaBall.

Follow me on X @BennyDaBall_OG !

Preservation record

Selected source bytes are verified against pinned metadata; the original eight graded mirrors also passed the previously recorded anonymous 1 MiB weight-prefix checks. This is not a full independent-device restore/load test. Hugging Face repository copies share a provider and do not guarantee permanent storage.

Collection · Source manifest · Original documentation