Qwen3.8-27B Uncensored HauhauCS Aggressive MTP — NVFP4 for NInfer

Native NInfer v2 artifact built for the Tesla V100 (sm_70) and derived from the HauhauCS uncensored finetune (HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF) of Qwen/Qwen3.8-27B. It includes Text, Vision, MTP, and frontend resources in a single file. No separate draft-model, vision tower, or tokenizer download is needed at inference time.

This file is specific to NInfer; it is not a Transformers checkpoint, Safetensors distribution, GGUF file, or generic NVFP4 interchange format.

Artifact

Stored tensor formats:

The nine w8_g32_fp16 objects are the NVFP4 identity's required endpoints: text/token_embedding, text/output_head, the five MTP projections (input_projection, attention/query_key_gate_value, attention/output, mlp/gate_up, mlp/down), and the two vision merger projections (merger/fc1, merger/fc2).

NVFP4 on Volta

NVFP4 here is a storage format, not a tensor-core instruction. The runtime decodes every stored value in software,

W[n,k] = decode_e2m1(code[n,k]) * decode_e4m3fn(scale[n,floor(k/16)]) / weight_global_scale

so a V100 with no FP4 hardware path serves the model at full precision of the quantized weights. No Blackwell-class GPU is required.

Requirements and download

• 64-bit Linux and two NVIDIA V100-SXM2-16GB GPUs (sm_70), tensor-parallel --tp 2;
• CUDA Toolkit 12.8 and the NVIDIA driver in the 570 branch;
• CMake 3.28+ and the dependencies described in the runtime README;
• the validated NInfer revision below or a compatible newer runtime.

Validation used two V100-SXM2-16GB with driver 570.211.01 and CUDA 12.8, Release/sm_70. Artifact size is not resident GPU memory: KV format/capacity, concurrency, Vision and CUDA Graphs affect memory use, and the 17.6 GB payload is sharded across both GPUs by tensor parallelism.

Build the validated runtime:

git clone https://github.com/tuxKOH/ninfer-V100X2.git
cd ninfer-V100X2
git checkout 685c09ab
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Download and verify the checksum:

hf download yinpeach/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4-NINFER \
  qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp-nvfp4.ninfer SHA256SUMS \
  --local-dir models/hauhaucs-nvfp4
(cd models/hauhaucs-nvfp4 && sha256sum --check SHA256SUMS)

Run

Tensor-parallel decode across both GPUs, with MTP speculation:

./build/apps/ninfer models/hauhaucs-nvfp4/qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp-nvfp4.ninfer \
  --prompt "The capital of France is" \
  --tp 2 --devices 0,1 \
  --max-context 4096 --max-new 256 --raw-output

OpenAI/Anthropic-compatible server with Vision enabled:

./build/apps/ninfer-serve models/hauhaucs-nvfp4/qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp-nvfp4.ninfer \
  --tp 2 --devices 0,1 \
  --host 127.0.0.1 --port 18080 \
  --max-context 8192 --kv-capacity 8192 \
  --prefill-chunk 1024 --vision

For ordinary decoding, omit the speculative options. Speculation is selected at startup, not by an OpenAI/Anthropic request field. Add --vision for image/video requests. Runtime KV storage choices are FP16 (default) and INT8 group-64; use --kv-dtype int8 for large context allocations.

NInfer serves one or more requests within a startup-fixed concurrency. It does not provide CPU/GPU offload or large-scale preemptive continuous batching.

Weight provenance and numerical boundaries

The text Linear weights come from the HauhauCS uncensored finetune, not from the official checkpoint. Its published GGUF uses Q4_K_P/Q8_K_P block formats that the NInfer GGUF lane does not decode, so the weights were dequantized to BF16 and re-quantized to NVFP4 by hand (E2M1 codes with E4M3FN block scales), one group-shared weight divisor per fused parent.

The activation and weight divisors, the Vision tower, the MTP module, and the w8_g32 endpoint source data are reused byte-for-byte from the QUASAR NVFP4 conversion (MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer). Fusing compatible source groups produced 256 NVFP4 parents — 32 attention, 96 GDN and 128 MLP. Fused groups share a single global-scale word; they were not rescaled after fusion.

Two classes of tensors were kept outside the NVFP4 path and stored as w8_g32 (8-bit, group-32, FP16 scales):

• the vocabulary endpoints text/token_embedding and text/output_head (248,320 × 5,120);
• the MTP projections and the two vision merger projections.

The vocabulary endpoints materialize from lm_head.weight; MTP and Vision originate from the QUASAR artifact and are unchanged. No tensor was substituted from the pre-existing QUASAR text tower; only divisors, Vision, MTP and endpoints are shared.

The 96 narrow GDN control projections (in_proj_a / in_proj_b, output width 48) cross the same BF16 boundary as in the reference conversion:

W[n,k] = decode_e2m1(code[n,k]) * decode_e4m3fn(scale[n,floor(k/16)]) / weight_global_scale

Represented values are rounded to BF16 and fused A-then-B into [96,5120] parents. These are decoded controls, not substitutions from the official BF16 checkpoint.

Validation and limits

The exact published file was used for integration validation on two V100-SXM2-16GB (driver 570.211.01, CUDA 12.8, Release/sm_70), tensor-parallel --tp 2. Selected checks passed:

• load and bind of Text, Vision and MTP resources;
• coherent greedy continuation on multiple prompts, including arithmetic and instruction-following;
• byte-identical greedy reruns on the same prompt (determinism);
• the shared MLP divisors, all 333 Vision tensors and all 12 MTP tensors are byte-identical to the FP8-mix sibling artifact; only layers 56–63 MLP and the two vocabulary endpoints differ.

Measured on this setup: decode ≈ 80–120 tok/s and prefill ≈ 1,280 tok/s at --tp 2. Throughput varies with prompt length, cache state and context capacity; these are single-workload spot checks, not a benchmark.

These checks do not establish broad reasoning/retrieval quality, full-corpus perplexity, or long-context accuracy. Raw-quantization error was characterized only through round-trip agreement; perplexity and long-context evaluation were not performed. Source-model quality scores are not measurements of this converted artifact.

Provenance

License and credit

Apache-2.0, matching the official Qwen3.8 and QUASAR source model cards. Credit belongs to Qwen for the base model, the HauhauCS authors for the uncensored finetune, the QUASAR authors for the NVFP4 divisors, Vision and MTP data, and NInfer and its contributors for conversion and inference.