Qwen3.8-27B-DSpark-NVFP4

NVFP4-quantized DSpark drafter for gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090.

Mixed-precision NVFP4 via NVIDIA ModelOpt: MLP and attention output projections at 4 bits, everything precision-sensitive left in BF16.

Retrained so the drafter sees what the model actually serves. The previous release was trained on a corpus with two mismatches against inference: tool examples used Hermes-style JSON (<tool_call>{"name":...}) while serving emits XML (<function=…><parameter=…>), and prompts were rendered with a generic template carrying no think blocks, no reasoning-effort preamble and no tools preamble. Both are now closed — every response is regenerated on-policy through the published chat template.

Same size, same recipe, same behaviour — the gain is purely in how often the target accepts a drafted block.

---

Results

RTX 5090 32 GB · concurrency 1 · 64K context · FP8 KV · 240 held-out prompts, identical across all profiles.

1.91× the decode throughput of the same checkpoint without speculation.

Throughput for the top two rows was measured cold in one session; the older rows come from earlier sessions, where run-to-run variance on this harness is roughly 2%.

Per domain

Acceptance is measured per request with the prefix cache flushed between requests, derived from cumulative verify counters rather than a sampled gauge.

Long-context is the one regression. The training corpus caps sequences at 2,048 tokens, so it was never well represented, and the shift toward agentic and think-block data appears to have cost it.

Agentic tool calling

Real tool use — tools declared, <tool_call> XML emitted, tool responses replayed — measured on 60 scenarios across 10 tool schemas. This is the workload the retraining targeted, and it improved in every phase:

Tool-call emission rate is unchanged (38/60 on both), so this is an acceptance gain rather than a change in what the model decides to do.

Note that the "JSON / structured output" row above is schema-constrained generation, not tool calling — those held-out prompts declare no tools. It was previously labelled "Tool calling", which understated the drafter on the workload it is named after.

---

Quantization recipe

Mixed precision, chosen by measuring every stage rather than quantizing everything:

Q/K/V are deliberately not quantized. A full-NVFP4 variant including them was built and measured: it reaches 150.73 tok/s but acceptance falls to 2.546. Keeping QKV in BF16 costs ~3 tok/s and buys back the acceptance, which is the better production trade.

Measured ladder:

Adding the O projection to MLP recovers acceptance fully; adding QKV loses it again.

fc (the 25600→5120 aux projection) stays BF16 because SGLang constructs it as a plain nn.Linear with no quantized loader.

Calibration

Post-training quantization with NVIDIA ModelOpt, calibrated to resemble real serving rather than synthetic input:

• 512 conversations, regenerated on-policy by the NVFP4 target at temperature 0
• Tool-weighted mixture (37.5% tool calling) to protect the most structurally rigid output
• Real captured target auxiliary features and real token embeddings for the draft input path
• Calibration data is disjoint from the evaluation set — no benchmark prompt was ever calibrated on

The tool weighting is measurable: it lifts tool acceptance from 3.15 to 3.85 versus a balanced calibration, at a cost of 0.8 tok/s overall.

---

Usage

Requires the Qwen3.8 SGLang build. Pin the image — DSpark runtime support moves quickly.

docker run --gpus all --ipc=host --shm-size 32g -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:qwen38-27b \
  sglang serve \
    --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
    --speculative-algorithm DSPARK \
    --speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \
    --speculative-draft-model-quantization modelopt_fp4 \
    --speculative-dspark-block-size 7 \
    --trust-remote-code --tp-size 1 \
    --context-length 122880 --kv-cache-dtype fp8_e4m3 \
    --attention-backend flashinfer --chunked-prefill-size 1024 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype bfloat16 \
    --max-mamba-cache-size 8 \
    --mm-feature-transport cpu \
    --cuda-graph-max-bs-decode 1 \
    --mem-fraction-static 0.86 --max-running-requests 1 \
    --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
    --host 0.0.0.0 --port 30000

Flags that are not optional on 32 GB:

• --mamba-ssm-dtype bfloat16 — without it the hybrid GDN state cache cannot allocate a single request slot and the server exits at startup.
• --max-mamba-cache-size 8 — this must leave headroom. Under DSpark, Qwen3.8 consumes 4 GDN state slots per request, so --max-mamba-cache-size 4 provisions exactly one request and nothing spare. Radix-cached prefixes hold slots from the same pool under extra_buffer_lazy, so if a cached state is locked when the next request needs a slot, eviction fails and the server aborts with AssertionError: Can not alloc mamba cache. Size 8 halves peak utilisation (0.75 → 0.38 under abort-heavy multi-turn traffic) at a cost of some KV pool. Rule of thumb: --max-mamba-cache-size ≥ 4 × --max-running-requests + 4.
• --speculative-draft-model-quantization modelopt_fp4 — this checkpoint is pre-quantized; loading it any other way will fail.
• --mem-fraction-static 0.86 — leave ~3.5 GB free. See below.

---

Context limits

The freed 1.31 GB converts directly into KV pool: 122,995 tokens at 64K settings, versus 72,485 for the BF16 drafter.

max_total_num_tokens is not the serving limit

At --mem-fraction-static 0.94 the server starts, reports a large pool, and then dies on the first long request:

causal_conv1d_triton.py:443   out = torch.empty_like(x)
torch.OutOfMemoryError: Tried to allocate 40.00 MiB. 9.81 MiB is free.

The pool is allocated optimistically and does not reserve for GDN prefill activations. Leave ~3.5 GB free and it is stable.

Concurrency and the 262K question

SGLang caps this configuration at max_running_requests = 1. Qwen3.8's GDN state cache costs ~147 MB per request across 4 slots; there is no room for a second on 32 GB. This ceiling is inherent to the target's hybrid architecture — the built-in MTP head hits it identically.

Full 262K with speculation does not fit on one 32 GB card even at 4 bits: the target's own FP8 KV needs ~8 GiB at 262K, and the draft KV pool is sized to the full target context. Use the no-speculation profile for maximum context.

Note also that speculative decoding accelerates decode only, never prefill.

Output quality is unchanged. The target verifies every drafted token under strict acceptance (speculative_accept_threshold_single=1.0, rejection sampling off). Speculation changes speed, not outputs — and that guarantee is what makes 4-bit drafting safe.

---

Model

Certified stack

DSpark runtime support is evolving quickly; this checkpoint was measured against exactly:

target   gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
draft    gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4
runtime  lmsysorg/sglang:qwen38-27b (0.0.0.dev0+qwen38.27b.g561c8f3)
hardware RTX 5090 32 GB

Evaluated on 240 held-out prompts (40 per domain), unseen during training and calibration. Greedy decoding, thinking disabled, 256 max new tokens. All profiles ran on identical prompts minutes apart on the same machine.

Caveat on margins. The acceptance figures average over many verification steps and are the more reliable signal. Throughput differences of 1–2% are measured across 72 requests per profile and should be read as directional.

License

Apache 2.0, matching the base drafter and the Qwen3.8-27B target.