Qwen3.8-27B-OBLITERATED — W4A16 (GPTQ, g128, symmetric)

4-bit GPTQ quantization of OBLITERATUS/Qwen3.8-27B-OBLITERATED, built to run on one 24 GB GPU (RTX 4090) with vLLM and the Marlin kernel.

19.45 GB · 81,920-token context on a single 4090 · vision tower kept · MTP head kept

Evaluation

MMLU, 0-shot, lm-eval with --limit 100 per subject (n = 5700) — the same protocol the upstream card reports — served through vLLM on one RTX 4090:

The two 4-bit builds are statistically indistinguishable (0.07 pp apart, ~1/7 of one standard error), and both sit ~0.7 pp under the bf16 card figure. Quantizing the gated-DeltaNet projections costs nothing measurable at this protocol.

Performance (1× RTX 4090, vLLM 0.28, fp8 KV, 73,728 ctx, --max-num-seqs 16)

Measured with vllm bench serve (random dataset, --ignore-eos). Combinations whose working set exceeds the 77,824-token KV pool are omitted — they cannot be co-resident and would only measure queueing.

PROMPT SCALING  (output 128 tok)
 prompt conc   tok/s  TTFT ms  p99TTFT TPOT ms  E2E ms
------------------------------------------------------
    128    1    52.3       85      152    18.6    2447
    128    2    94.5      138      144    20.2    2708
    128    4   176.4      205      231    21.2    2901
    128    8   326.7      380      422    21.7    3133
    128   16   456.9      953     3980    24.9    4120
    512    1    49.9      199      206    18.6    2563
    512    2    86.5      341      392    20.6    2958
    512    4   150.1      670      739    21.6    3410
    512    8   245.6     1067     1425    24.4    4161
    512   16   314.8     1783     6030    34.2    6124
   2048    1    41.4      719      729    18.7    3088
   2048    2    86.4      328      383    20.7    2961
   2048    4   114.7     1137     2766    26.1    4449
   2048    8   123.4     1874     5570    50.3    8263
   2048   16   127.8     6799    13874    66.3   15223
   8192    1    24.1     2907     2923    18.9    5311
   8192    2    30.5     4256     5698    32.5    8382
   8192    4    36.0     5479    11274    68.5   14183
   8192    8    37.4    13160    25507   104.1   26376
  16384    1    15.2     6006     6018    19.2    8441
  16384    2    17.5     8135    11907    51.0   14610
  16384    4    18.4    13142    25833   106.6   26686
  32768    1     8.3    12983    12997    19.7   15482
  32768    2     8.9    17353    25718    88.1   28538
  65536    1     4.0    29761    29768    20.7   32387
  72000    1     3.5    33534    33563    20.9   36182

LONG GENERATION
 prompt   out conc   tok/s  TTFT ms TPOT ms   E2E s
---------------------------------------------------
    512  1024    1    53.1      199    18.6    19.3
    512  1024    2    96.5      342    20.4    21.2
    512  1024    4   182.1      669    21.3    22.5
    512  1024    8   344.9     1069    22.2    23.7
    512  1024   16   494.7     4500    25.2    30.3
    512  2048    1    53.3      200    18.7    38.4
    512  2048    2    97.0      345    20.5    42.2
    512  2048    4   184.0      672    21.4    44.5
    512  2048    8   348.9     1045    22.4    46.9
    512  2048   16   422.5     9243    27.6    65.8
   8192  2048    1    48.9     2914    19.0    41.8
   8192  2048    4   149.6     4603    24.5    54.7
  32768  2048    1    38.2    12978    19.9    53.6
    512  4096    1    53.3      201    18.7    76.8
    512  4096    2    96.9      343    20.6    84.5
    512  4096    4   184.2      673    21.6    89.0
    512  4096    8   346.8      931    22.8    94.4

Reading this:

• Decode is memory-bandwidth-bound and essentially free of prompt length. TPOT sits at ~18.6 ms/token at concurrency 1 for every prompt size from 128 to 72,000. What grows with prompt length is TTFT (prefill): 85 ms at 128 tokens, 33.5 s at 72,000.
• Long outputs cost nothing extra per token. A 4096-token generation sustains the same 53.3 tok/s as a 128-token one. Generation length affects total wall time linearly and nothing else.
• Concurrency scales well only while prefill is cheap. At 128-token prompts, 16 concurrent gives 8.7× the single-stream throughput. At 2048+ prompts it saturates by 4–8 streams as prefill dominates, and p99 TTFT degrades sharply (13.9 s at 2048/16).
• Peak observed throughput: 494.7 tok/s (prompt 512, 1024 output, 16 concurrent).
• The largest prompt that fits is 72,000 tokens, not 73,600 — the chat template's wrapper tokens count against --max-model-len, and exceeding it returns HTTP 400.

Long-generation quality

Greedy (temp 0, repetition_penalty 1.15) generations of up to 3072 tokens across code, prose and reasoning prompts:

Quantizing the gated-DeltaNet layers does not produce the drift/looping failure mode at this length. One caveat worth knowing: the single generation that ran into the token cap had drifted into enumerating invented HTTP headers — semantic drift rather than repetition, the normal consequence of an unbounded greedy continuation. Keep max_tokens bounded to the task.

Sub-4-bit variants — tested, not shipped

To see whether more context could be had on the same card, two lower-precision builds were made with the identical recipe and calibration set and evaluated the same way. Both work and both are fast, but both cost 3–4 pp of MMLU, so this 4-bit build remains the recommended one.

By group (4-bit → mixed → W3): humanities 82.2 → 79.0 → 77.7 · other 82.1 → 77.8 → 79.5 · social sciences 87.3 → 85.4 → 84.2 · STEM 77.4 → 74.8 → 73.0. The loss is general, not confined to one subject.

What the experiment established:

• 3-bit is not slow on Ampere/Ada. Marlin rejects uint3b4 on sm89 and the legacy GPTQ path is 4/8-bit only, so vLLM falls back to HummingLinearKernel — but decode is memory-bandwidth-bound, and measured decode on Humming (forced onto this very checkpoint) matches Marlin within 3 %. Fewer bits per weight therefore make the 3-bit builds decode faster than this one. Only prefill pays: ~40 % higher TTFT.
• The quality loss is in the MLP, not the attention paths. The mixed build kept attention and the 48 gated-DeltaNet layers at 4-bit on the theory that recurrent state would be fragile; it recovered only 0.8 pp of the 3.7 pp loss. Since the MLP is 70 % of the quantized weights, there is no split that keeps most of the memory saving without most of the quality cost.
• The bf16 floor. lm_head (2.37 GiB — vLLM's ParallelLMHead cannot load a packed head), embed_tokens (2.37 GiB) and the vision tower (0.86 GiB) cannot be quantized and together are a third of the 3-bit model. Going below 3 bits on the linears buys little further context.
• System-RAM offload reaches 262 K but at 3 tok/s. --cpu-offload-gb / --offload-backend prefetch stream hot decoder weights over PCIe every token (16–20× slower). Offload applies only to decoder layers, so the cheap-to-move tensors (embeddings, vision) cannot be offloaded. CPU KV offload does not raise max-model-len at all.

The sub-4-bit checkpoints are kept locally for workloads that value context over accuracy (long-document retrieval, log analysis); they are not published.

What is and isn't quantized

This model is a hybrid: 64 decoder layers = 48 gated-DeltaNet (linear attention) + 16 full attention, plus a vision tower and a multi-token-prediction head.

Quantized to int4:

• all MLP gate_proj / up_proj / down_proj (64 layers — the bulk of the parameters)
• full-attention q_proj / k_proj / v_proj / o_proj (16 layers)
• gated-DeltaNet in_proj_qkv / in_proj_z / out_proj (48 layers, ~5.5 B params)

Kept in bf16:

Quantizing the DeltaNet projections is what makes this fit 24 GB. Builds that keep all linear attention in 16-bit land at ~25–30 GB and need two cards.

Serving (vLLM)

vllm serve <this-repo> \
  --served-model-name qwen3.8-27b-obliterated \
  --dtype bfloat16 \
  --max-model-len 81920 \
  --max-num-seqs 2 \
  --gpu-memory-utilization 0.97 \
  --kv-cache-dtype fp8 \
  --port 8000

Confirm Using MarlinLinearKernel for CompressedTensorsWNA16 in the startup log.

Measured on one RTX 4090 (23.99 GiB usable): weights 17.45 GiB, KV cache 2.85 GiB = 81,920 tokens at fp8. Adding --language-model-only frees the ~0.9 GiB vision tower for more KV if you never send images.

Concurrency is limited by the DeltaNet state, not by KV. Each running sequence holds an fp32 recurrent state across the 48 linear-attention layers; vLLM allocates a fixed pool of "Mamba cache blocks" and refuses to start if --max-num-seqs exceeds it. At these settings the ceiling is ~46 sequences.

Sampling — these matter

Per the upstream card: temperature 0, repetition_penalty 1.15, no system prompt, thinking off, max_tokens ≥ 2048.

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3.8-27b-obliterated",
  "messages": [{"role": "user", "content": "..."}],
  "temperature": 0, "max_tokens": 2048, "repetition_penalty": 1.15,
  "chat_template_kwargs": {"enable_thinking": false}
}'

enable_thinking: false is not optional in practice. The bundled template only prefills the empty thinking block when thinking is off; with it on, raw reasoning prose is emitted directly into content (vLLM has no reasoning parser configured for this family).

Speculative decoding (MTP)

The MTP head ships in model-mtp.safetensors in bf16 and is in quantization_config.ignore.

  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

It costs ~0.9 GiB resident, so lower --max-model-len accordingly. Output at temperature 0 is unchanged (every draft token is verified against the quantized target) — MTP only affects speed. Note the head is stock, not abliterated, so acceptance is lower on prompts where this model diverges from stock Qwen.

Notes / gotchas found while building this

• lm_head cannot be quantized for vLLM here. Quantizing it saves ~1.9 GiB and would buy roughly 50 K more context, but vLLM's ParallelLMHead only binds lm_head.weight: ValueError: There is no module or parameter named 'lm_head.weight_packed'. The head in this repo is the original bf16 tensor copied from the source, not a dequantized one, so there is no round-trip error.
• The MTP head must be re-attached manually. transformers' Qwen3_5ForConditionalGeneration does not model the mtp.* submodule, so from_pretrained loads 1184 of the source's 1199 tensors and save_pretrained silently drops all 15 MTP tensors. They are copied back verbatim here.
• The upstream chat template drops tool definitions. It handles only system/user/ assistant, so the tool role and tool schemas are discarded with no error — tool calling never fires in an agent harness. That is a property of the upstream weights, not of this quantization. This repo ships the upstream template unchanged; if you need tool calling, substitute the stock Qwen3.8 template (the tokenizers are equivalent).

Credit

Base weights: OBLITERATUS/Qwen3.8-27B-OBLITERATED (Apache-2.0). The choice of which DeltaNet projections to keep in bf16 follows MurtazaNasir/Qwen3.8-27B-OBLITERATED-AWQ-W4A16, an earlier AWQ build of the same model; this repo differs in using GPTQ with symmetric quantization (no zero-points), which is what yields the larger KV budget.