Qwen3.8-27B for NInfer

This model card is the version-controlled source for neroued/Qwen3.8-27B-NInfer.

The repository contains Qwen3.8-27B converted to the native NInfer .ninfer artifact format. The artifact is intended only for NInfer; it is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Artifact

The Text body uses Q4/Q5 projections, while the token embedding and full output head use q8_g32_fp16. The file also contains Vision, MTP, DFlash2, the optimized proposal head and frontend resources. Vision and speculative weights are loaded only when selected at startup.

Verify a downloaded file with:

printf '%s  %s\n' \
  '81f924d440c27261d820c19a9f8d45794c5aee410f8a68bd358133fa8c0375da' \
  'qwen3_8_27b.ninfer' | sha256sum --check

This release includes the complete DFlash2 companion weights from z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4. Select --spec dflash2 --draft-tokens 7 --proposal-head optimized; draft counts 1..15 are supported. Use the current runtime for DFlash2. Existing performance and evaluation tables retain their stated MTP configurations and revisions.

Requirements

• Current NInfer checkout, built from source;
• 64-bit Linux;
• NVIDIA GeForce RTX 5090 (sm_120a);
• Build dependencies from the build guide.

Already have the official v2 file? Upgrade it locally without downloading the weights again.

NInfer does not provide an install target or packaged binary. See the repository README for source-build dependencies.

Download and run a CLI example

hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

./build/rtx5090-release/apps/ninfer models/qwen3_8_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 32768 \
  --max-tokens 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --proposal-head optimized

For images, videos, and structured chat history, see the CLI guide.

Start a local server

./build/rtx5090-release/apps/ninfer-serve models/qwen3_8_27b.ninfer \
  --host 127.0.0.1 \
  --port 8080 \
  --max-context 240000 \
  --context-capacity auto \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --proposal-head optimized \
  --preserve-thinking

Each request has a 240,000-token logical ceiling. KV histories and continuation state share the Device Context byte capacity selected at startup. The log reports the pool size and estimated token capacity. Requests allocate as they advance; resource pressure can reclaim retained prefixes or pause requests for later restoration/replay. See the startup reference for capacity, Host cache and deployment controls.

See the HTTP serving guide for the API surface and the resource scheduling reference for cache and admission semantics.

Supported use

The artifact supports:

• text generation in thinking and non-thinking modes;
• image, multi-image, video, and mixed multimodal messages;
• MTP speculative decoding with draft windows from one to five;
• DFlash2 with draft windows from one to fifteen using the included DFlash2 companion weights (--spec dflash2 --draft-tokens 7, optionally --proposal-head optimized);
• BF16, INT8, FP8, NVFP4, and K8V4 KV cache;
• CUDA Graph decode and compatible-prefix reuse;
• startup-bounded small-scale concurrent serving with true batched decode;
• the NInfer CLI;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages serving.

Evaluation

The artifact was evaluated through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and INT8 group-64 KV. EvalScope 1.9.0 used 0-shot prompts, rule-based scoring, and one sample per problem with temperature 1.0, top-p 0.95, top-k 20, presence penalty 0.0, and seed 42. The text suite ran at a 262,144-token context limit; the multimodal suite ran with --vision at a 81,920-token limit.

All 1,723 configured samples completed and were scored. IFBench additionally reports 81.00% instruction-level strict, 80.67% prompt-level loose, and 83.67% instruction-level loose. These are single-sample results, not pass@k.

The official Qwen3.8-27B BF16 figures come from the upstream model card; its sampling settings and IFBench metric level are not stated there, so the last column is not a same-protocol comparison. The groupwise-int deltas stay within ±3.1 points on the four overlapping benchmarks, and the upstream card reports no AIME results.

Limits

• This artifact targets one RTX 5090, with one to eight active requests per Engine.
• KV and continuation state share the startup Context pool. Usable context depends on the selected KV type, enabled components, concurrency and available memory.
• NInfer returns generated tool calls to the client; it does not execute them.

Provenance

The local source configuration, tensor index, frontend resources, and published CRC32 inventory were checked against the canonical Hugging Face source revision above before conversion publication.

The artifact identity, summarized object inventory, and conversion provenance are published in artifact-manifest.json. The exact storage contract is maintained in the v3 container reference.

License

This NInfer artifact is distributed under the Apache License 2.0. The source Qwen3.8-27B repository is also licensed under Apache-2.0. Users remain responsible for complying with the license and applicable laws.