Nex-N2.5-mini FP8 W8A8 with Calibrated FP8 KV Scales

This repository contains an FP8 quantized derivative of nex-agi/Nex-N2.5-mini, pinned to source revision 87420286149d9cce9bd46cd335ef9bda33c37c1b.

The release uses block-scaled FP8 W8A8 for eligible linear layers and includes calibrated static FP8 KV-cache scales. The multimodal vision tower, hybrid linear-attention state, routing components, embeddings, output head, and other sensitive modules remain in their source precision.

This is a community quantization. Refer to the original Nex-N2.5-mini model card for the model family description, benchmark results, intended applications, and upstream usage guidance.

Artifact summary

• Architecture: Qwen3_5MoeForConditionalGeneration
• Model type: qwen3_5_moe
• Total parameters recorded in the weight index: 35,109,233,796
• Native maximum sequence length: 262,144 tokens
• Transformer layers: 40
• Routed experts: 256 per MoE layer
• Active experts: 8 per token
• Weight layout: two indexed safetensors shards
• Tensor storage: 36,601,121,032 bytes, approximately 34.1 GiB
• Quantization format: Compressed Tensors 0.18.0
• Included assets: tokenizer, processor configuration, chat template, generation configuration, quantization recipe, and validation report

Quantization

The model was quantized with LLM Compressor using the FP8_BLOCK W8A8 scheme.

Weights

• Eligible Linear weights use static, symmetric 8-bit floating-point quantization.
• Weight scales use 128 x 128 blocks.
• The artifact contains 30,970 FP8 weight tensors and 30,970 corresponding weight-scale tensors.

Activations

• Input activations use dynamic, symmetric 8-bit floating-point quantization.
• Activation scaling is grouped with a group size of 128.

KV cache

• K and V cache scales were calibrated as static, symmetric, tensor-wise FP8 scales.
• The artifact contains 10 K-cache scales and 10 V-cache scales.
• Calibration used 512 samples from HuggingFaceH4/ultrachat_200k, with a maximum calibration sequence length of 2,048 tokens.
• A runtime must be configured to use an FP8 KV cache for these scales to take effect.

Components retained at source precision

The quantization recipe excludes components that are sensitive, unsupported, or intentionally preserved:

• multimodal vision tower;
• token embeddings and language-model output head;
• hybrid linear-attention convolution and projection state;
• MoE routers and shared-expert gates;
• MTP and hypernetwork components.

Validation found 595 protected tensors that exactly match the source checkpoint.

Validation

The finalized artifact passed the following checks:

• both indexed safetensors shards open successfully;
• all 62,656 indexed tensors are present;
• architecture and model type match the source derivative;
• 30,970 FP8 weights have matching scale tensors;
• calibrated K-cache and V-cache scales are present;
• 595 protected tensors match the BF16 source exactly;
• representative routed-expert tensors remain numerically consistent with their source weights after FP8 dequantization;
• tokenizer, processor configuration, chat template, generation configuration, and quantization metadata are present;
• the model loads and generates coherent text through an OpenAI-compatible vLLM server;
• a multi-turn agentic acceptance test completed 27 model turns and 52 tool calls without a model or request error. It produced the requested utility and reports. One of eight evaluator checks failed because the generated JSON used a different field name than the evaluator expected.

The vision weights were preserved exactly, but this release has not been given a separate end-to-end image benchmark.

Serving with vLLM

Use a recent vLLM build with support for Qwen3.5 MoE, Compressed Tensors FP8_BLOCK, multimodal input, and FP8 KV cache.

vllm serve cbert33/Nex-N2.5-mini-FP8-Calibrated \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Memory use, concurrency, and maximum practical context depend on the serving engine, hardware, KV-cache allocation, and enabled multimodal features.

Sampling and thinking modes

The upstream model card recommends:

• temperature: 0.7
• top_p: 0.95
• top_k: 40

The included chat template uses reasoning_effort:

• none: respond without a reasoning trace;
• medium: adaptive thinking, the upstream default;
• high: always enable thinking.

Example OpenAI-compatible request:

{
  "model": "Nex-N2.5-mini-FP8-Calibrated",
  "messages": [
    {
      "role": "user",
      "content": "Explain how binary search works."
    }
  ],
  "reasoning_effort": "medium",
  "temperature": 0.7,
  "top_p": 0.95
}

The request model name must match the name exposed by your serving engine.

Function calling

The source model uses the qwen3_coder tool-call parser. Enable automatic tool choice and configure that parser in the serving engine. The included chat template should be used unless the runtime supplies a verified equivalent.

Limitations

• Quantization can change output quality relative to the BF16 source.
• Runtime support for this architecture and quantization layout varies by engine and version.
• FP8 KV scales are used only when the runtime enables an FP8 KV cache.
• The local validation described above is an acceptance check, not a comprehensive benchmark suite.
• Follow the upstream model card for the original model's capabilities, safety considerations, and known limitations.

License and attribution

The source model is released under the Apache 2.0 license. This derivative retains that license. Nex-N2.5-mini was created by Nex-AGI.