Qwen3.5-9B for NInfer

This model card is the version-controlled source for ruwwww/qwen3.5-9b-ninfer.

The repository contains Qwen3.5-9B converted to the native NInfer .ninfer artifact format. The artifact is intended only for NInfer; it is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Artifact

The file contains the registered Text, Vision, MTP, proposal-head, tokenizer, chat-template, generation, and media-processor objects required by NInfer.

Verify a downloaded file with:

printf '%s  %s\n' \
  '5e823ea5b4df7c75c630cb5ff90017cf17769c68bbf9545640cf981af5ec7bd6' \
  'qwen3_5_9b.ninfer' | sha256sum --check

Source model

Qwen3.5-9B is a multimodal dense model with a mixed linear/full-attention text backbone:

Requirements

• NInfer revision 24bd04d or later, built from source;
• 64-bit Linux;
• NVIDIA GeForce RTX 5090 (sm_120a);
• CUDA Toolkit 13.1 or newer.

NInfer does not provide an install target or packaged binary. See the repository README for source-build dependencies.

Download and run

hf download ruwwww/qwen3.5-9b-ninfer \
  qwen3_5_9b.ninfer \
  --local-dir models

./build/apps/ninfer models/qwen3_5_9b.ninfer \
  --prompt "What is the capital of France?" \
  --max-context 4096 \
  --max-new 256

Serve it through the OpenAI-compatible HTTP endpoint:

./build/apps/ninfer-serve models/qwen3_5_9b.ninfer \
  --model-id qwen3.5-9b \
  --max-context 16384 \
  --spec mtp --draft-tokens 3

For images, videos, structured chat history, and HTTP serving, see the NInfer documentation.

Supported use

The artifact supports:

• text generation in thinking and non-thinking modes;
• image, multi-image, video, and mixed multimodal messages;
• MTP speculative decoding with draft windows from one to five;
• BF16 and INT8 group-64 KV cache;
• CUDA Graph decode and compatible-prefix reuse;
• the NInfer CLI;
• OpenAI Chat Completions and Anthropic Messages serving.

Verified behavior

The 9B artifact was verified against the source BF16 model (Hugging Face Transformers on an RTX 5090):

• Prefill activations match the reference top-10 logits exactly (byte-identical per step).
• Greedy decode tokens 0-7 are identical to the reference; later steps diverge only on near-tie logits within quantization drift, with both streams remaining coherent.
• Observed serving throughput on one RTX 5090 (greedy, short prompts): ~793 tok/s prefill and ~241 tok/s decode without speculation; with MTP=3, ~445-455 tok/s decode and ~84% draft acceptance.

Limits

• The artifact is accepted only by NInfer revision 24bd04d or later and the matching registered target.
• NInfer currently executes on one RTX 5090, one CUDA device, and one active request per Engine.
• It does not provide continuous batching, multi-GPU execution, CPU/GPU offload, or distributed serving.
• Context allocation is subject to GPU memory and the selected KV-cache type.
• NInfer does not execute generated tool calls.

Provenance

License

This NInfer artifact is distributed under the Apache License 2.0. The source Qwen3.5-9B repository is also licensed under Apache-2.0. Users remain responsible for complying with the license and applicable laws.