Naive-N0.5-Flash NVFP4 — vLLM on four RTX PRO 6000 Blackwell GPUs

Original model by Naive AI. This release is built on their work.

The Naive AI team created and openly released Naive-N0.5-Flash, including its sparse-attention adaptation, continued pretraining, model weights and reference implementation. Their research and engineering are the foundation of this repository. Many thanks to the team for sharing the model, code and technical details with the community and making derivatives like this possible.

Please support and credit the original project: original model · GitHub · technical report and research · Naive AI.

Naive's model in turn builds on Xiaomi MiMo-V2.5-Base and the DeepSeek team's sparse-attention research. Credit for that foundation belongs to those teams as well. This repository is an unofficial community quantization and serving adaptation: its contribution is the expert-only NVFP4 conversion, validation, and native vLLM integration for RTX PRO Blackwell hardware.

NVFP4 expert quantization of NaiveAI/Naive-N0.5-Flash, a sparse-attention derivative of Xiaomi MiMo-V2.5. The model has approximately 309B total parameters and 15.5B active parameters per token. The 49 weight shards occupy 182.57 GB (170.03 GiB).

Serve this checkpoint with the included native vLLM model plugin and Docker recipe below. The tested engine is vLLM 0.28.0, with tensor parallelism across four 96 GB RTX PRO 6000 Blackwell GPUs. vLLM runs scheduling, paged KV-cache allocation, sampling, prefix caching, chunked prefill and the streaming OpenAI-compatible API. The plugin supplies Naive's model implementation and NVFP4 expert kernels.

Naive's architecture requires this plugin; the pinned stock vLLM image does not implement it. Installing the plugin is part of the Docker build. This repository does not provide a tested SGLang implementation. Keep the original naive_n05_flash architecture in config.json.

Current serving recipe: v2

The current v2 release uses the compact-KV plugin and the configuration in compose.yml. Download that tagged revision so the runtime and recipe match the checkpoint. The weights are unchanged from v1.

The fixed KV allocation takes precedence over --gpu-memory-utilization. Four slots allow four active requests when their combined cache use fits; they do not reserve four full-length contexts. The v2 validation includes an exact retrieval from a 1,019,986-token prompt and four simultaneous short requests. The 8K benchmarks later in this card belong to the historical v1 release.

Run with vLLM

The tested machine uses four NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition 96 GB GPUs (SM120), a Threadripper PRO 7955WX, 498 GiB host RAM, Linux and driver 595.91.07. Have Docker Compose and NVIDIA Container Toolkit working, with all four GPUs visible to containers. Allow roughly 250 GB of free disk for the checkpoint, container images and compilation caches. This recipe is validated on the four-GPU machine described here; smaller configurations have not been validated.

Download the release using the Hugging Face CLI:

hf download piotrkosecki/Naive-N0.5-Flash-NVFP4 \
  --revision nvfp4-vllm-v2 \
  --local-dir ./Naive-N0.5-Flash-NVFP4
cd Naive-N0.5-Flash-NVFP4
sha256sum --check SHA256SUMS

Build the pinned runtime, check the checkpoint and hardware, and start vLLM:

export LOCAL_UID="$(id -u)"
export LOCAL_GID="$(id -g)"
mkdir -p runtime-cache
docker compose build naive-vllm
docker compose run --rm --no-deps --entrypoint python3 naive-vllm \
  /opt/naive/scripts/preflight.py
docker compose up -d naive-vllm
docker compose logs -f naive-vllm

Free the GPUs from other large models before starting. This recipe reserves 30 GiB of KV cache per GPU in addition to weights and runtime workspaces; monitor actual memory use, especially on a GPU driving a display. Keep runtime-cache: the first start and the first request of a new shape can compile kernels and take substantially longer. Cached starts still need model loading and graph capture. Wait for the API server to become ready:

curl --fail http://127.0.0.1:8000/health
curl --fail http://127.0.0.1:8000/version
curl --fail http://127.0.0.1:8000/v1/models

/version reports 0.28.0; /v1/models lists Naive-N0.5-Flash-NVFP4. The API binds to localhost, and container restart is manual. Stop it with docker compose stop naive-vllm.

The image entrypoint is python3 -m vllm.entrypoints.openai.api_server. The full launch configuration is in compose.yml; its vLLM arguments are:

--model /models/Naive-N0.5-Flash-NVFP4 \
--served-model-name Naive-N0.5-Flash-NVFP4 \
--trust-remote-code \
--tensor-parallel-size 4 --distributed-executor-backend mp \
--dtype bfloat16 --max-model-len 1048576 \
--max-num-batched-tokens 2048 --max-num-seqs 4 \
--kv-cache-memory-bytes 32212254720 \
--disable-custom-all-reduce \
--enable-prefix-caching --enable-chunked-prefill \
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1],"cudagraph_num_of_warmups":2}' \
--enable-auto-tool-choice --tool-call-parser mimo --reasoning-parser mimo \
--host 127.0.0.1 --port 8000

Send requests

Streaming chat:

curl -N http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Naive-N0.5-Flash-NVFP4",
    "messages": [{"role": "user", "content": "Explain how a hash table works."}],
    "temperature": 0,
    "max_tokens": 256,
    "chat_template_kwargs": {"enable_thinking": false},
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

For an OpenAI-compatible client, use base URL http://127.0.0.1:8000/v1 and model Naive-N0.5-Flash-NVFP4. No API key is configured; clients that require a value can use a placeholder.

For thinking, set chat_template_kwargs to {"enable_thinking": true} and allow enough output tokens. This vLLM version returns separate message.reasoning and message.content fields. Streaming responses use delta.reasoning and delta.content.

Tool calls use the normal OpenAI tools schema, for example:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Naive-N0.5-Flash-NVFP4",
    "messages": [{"role": "user", "content": "Call get_weather for Warsaw."}],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get weather for a city",
      "parameters": {"type": "object", "properties": {
        "city": {"type": "string"}
      }, "required": ["city"]}
    }}],
    "tool_choice": "auto",
    "temperature": 0,
    "max_tokens": 192,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Release v2: compact KV and 1M configuration

This revision incorporates the compact-KV plugin contribution by mmssix. The model weights and quantization are unchanged. The plugin now shares FlashInfer MoE workspaces across layers, uses vLLM sliding-window allocation for SWA, stores BF16 K/V and compact FP8 indexer payloads, and bounds long-context sparse-attention work. The Compose recipe configures the source model's 1,048,576-token maximum context, a 30 GiB per-GPU KV budget, 2,048-token prefill chunks and four scheduler slots. The fixed KV budget takes precedence over automatic sizing by --gpu-memory-utilization; four slots do not imply that four full-length prompts fit simultaneously.

On October 4, 2026, this v2 build passed local preflight, short chat, four simultaneous chat requests, a tool call, and a synthetic 1,019,986-prompt-token retrieval on the four-GPU workstation. A code placed halfway through that prompt was returned exactly; the long request completed in 418.1 seconds, with no preemptions or out-of-memory failure. vLLM reported a 2,740,779-token KV pool. The v2 acceptance record contains the exact scope and results. This is one long-context retrieval check, not a broad quality evaluation.

The contributor separately reports 1.03M-token needle retrieval at multiple depths and perplexity comparisons through 28K tokens; those results are not reproduced in the historical validation files below. Sparse selection now uses topk, so exact-score ties can choose different keys from the v1 stable argsort. Floating-point score accumulation also changes; output is not promised to be token-for-token identical to v1. Full-context prefill can take several minutes. The v1 release remains available at tag nvfp4-vllm-v1.

Historical v1 performance and validation

Both benchmark runs below were measured on September 28, 2026 with the original v1 release build on this workstation. They are retained as historical baseline measurements, not v2 performance claims:

The tokens/sec figures describe generation by the full model across this four-GPU workstation.

A second run measured 55.00–55.11 tokens/sec during decode. Both runs exceeded 50 tokens/sec in all five cases. Raw responses, token IDs, timestamps and usage counts are in run 1 and run 2.

These v1 measurements use the full model in native vLLM, one active request, temperature 0 and 256 generated tokens per response. The client records token-bearing SSE events and checks the exact returned token IDs against vLLM's usage count. Decode rate is the number of tokens after the first event divided by the time from the first to the last token event. Prompt processing and time to first token are reported separately. Prefix caching is enabled; the long prompts share background text, and repeated prompts can reuse cached prefixes. The time-to-first-token column is not a cold-prefill benchmark. Compilation and prompt processing affect end-to-end speed.

CUDA graphs currently cover one active decode request. Multiple active requests run through the eager path and are slower per request. Four simultaneous requests passed a functional test; the single-request speeds above do not describe concurrency throughput.

Run the API checks on v2 while no other client is using the server; the benchmark script can collect a new v2 result:

python3 runtime/scripts/validate_api.py
python3 runtime/scripts/benchmark_stream.py \
  --tokens 256 --cases 5 --output native-vllm-benchmark.json

The historical v1 validation includes full-model arithmetic, geography, Python code, structured tool calls, reasoning, longer-context generation and four concurrent requests. Tiny-model tests compare 144 greedy tokens exactly with the quantized reference while exercising graph replay, batched scheduling, chunked prefill and prefix caching. A real expert layer was also checked with tensor-parallel slicing. The older reports in validation/ describe v1 checks; the new v2 acceptance record above covers the updated runtime. These are smoke and implementation checks, not an evaluation of accuracy loss versus BF16 on a benchmark suite.

Runtime limits and implementation

• Configured maximum context: 1,048,576 total input plus output tokens. The v1 benchmark above tested up to 8,192 tokens; local v2 acceptance tested a 1,019,986-token prompt, and the contributor reports separate retrieval probes through about 1.03M tokens.
• TP4, pipeline parallelism 1; 2,048-token prefill chunks; up to four scheduled sequences; 30 GiB fixed cache budget per GPU. The shared cache can fill before all four requests reach the configured maximum length, causing preemption and recomputation. K/V are BF16; the indexer key is stored as FP8 E4M3 payload with a separate FP32 scale. KV cache is not NVFP4.
• Single-token execution fuses RMS normalization, Q/K/V projection and rotary embedding operations, while retaining source BF16 rounding points.
• The plugin preserves the source sliding window of 128, sparse selection of up to 2,048 tokens and attention value scaling. Sparse-score arithmetic and exact-tie selection can differ from v1; verify quality for your workload.
• NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are required by the tested workstation's failed direct peer-copy integrity checks. The recipe keeps those settings and uses NCCL's host transport.
• FlashInfer 0.7.0's direct-micro expert path returned zeros for this checkpoint's TP4 shape. The plugin disables that path and uses its validated regular MMA micro/static/dynamic kernels. This workaround depends on the pinned FlashInfer version.
• sitecustomize.py isolates FlashInfer 0.7 source-built kernels from the base image's older prebuilt cubins. Keep the supplied dependency pins and persistent cache directories.

The Dockerfile pins the public vLLM base image by digest. Tested versions: vLLM 0.28.0, PyTorch 2.13.0+cu130, Transformers 5.17.0, FlashInfer 0.7.0, CUTLASS DSL 4.6.2 and Triton 3.7.1. The plugin uses vLLM's model registration mechanism and is included as source in runtime/vllm_naive/.

Quantization and provenance

Source revision: 0235b3b5ff27422b1f57cdc2acddfaf643e08356. NVIDIA Model Optimizer revision: 23355eda90a25c290f9b1fdfb928ad54caae7d10.

Only routed experts are quantized: NVFP4 E2M1 weights and activations, block size 16, FP8 block scales and FP32 global scales. Attention, indexer, router, embeddings and output head retain source precision. All 472 non-expert tensors match source shapes, dtypes and payload hashes. There are 36,096 packed expert matrices and 144,856 exported tensors; total tensor payload is 182,549,965,184 bytes.

Layerwise max calibration used 128 sequences of 4,096 tokens (524,288 total), equally divided between Magicoder OSS Instruct and UltraChat, with seed 20260928. Exact dataset revisions and calibration hash are in quantization_provenance.json. Layer 47's router was materialized in BF16 during calibration; the final router was restored byte-for-byte from the FP32 source. An exporter naming issue required reconstructing the unchanged embedding, final norm and output-head tail; all 48 calibrated layer shards were retained and the complete export was subsequently audited.

The source model's MIT license is preserved in LICENSE. Upstream configuration/modeling files retain their Apache-2.0 headers. Additional runtime code is MIT licensed; see runtime/THIRD_PARTY_NOTICES.md.

Acknowledgments and citation

This release depends on substantial work by the original model authors and the open-source inference community:

• Naive AI / NaiveAI Team — creation and release of Naive-N0.5-Flash, the hybrid SWA/DSA adaptation, continued pretraining, checkpoint, chat template, model implementation and research documentation. Please cite their work when using this derivative.
• Xiaomi MiMo — the MiMo-V2.5 base model on which Naive-N0.5-Flash builds.
• DeepSeek — the DeepSeek Sparse Attention research underpinning Naive's sparse-attention design.
• vLLM and FlashInfer — the serving engine and Blackwell NVFP4 expert kernels used in this deployment.
• mmssix — compact-KV, length-aware sparse attention, and shared-workspace plugin patch contributed in the release discussion.
• NVIDIA Model Optimizer and CUTLASS — quantization tooling and GPU kernel infrastructure.
• Hugging Face Transformers, PyTorch and Triton — model-loading, tensor-computation and kernel-building infrastructure.

The original weights and their capabilities come from the upstream model teams. This quantized release retains Naive AI's model identity, copyright notices and license. Please direct recognition for the model and its research to the original authors.

The following is the citation supplied by Naive AI in the original model card:

@misc{naiveai2026naiven05flash,
  title  = {Naive-N0.5-Flash: Building Frontier AI with AI},
  author = {{NaiveAI Team}},
  year   = {2026},
  url    = {https://naive.ai/en/research/}
}