AxionML Laguna-S-2.1-NVFP4

Mirrored by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.

Quantized by Poolside. The weights in this repository are an unmodified copy of poolside/Laguna-S-2.1-NVFP4 (revision 826aacdf6d8b2699d4e367def6f17c83b06044c2). All credit for the quantization belongs to Poolside.

This is an NVFP4-quantized version of poolside/Laguna-S-2.1 (117.6B total parameters, 8.5B activated), quantized with NVIDIA Model Optimizer.

About NVFP4 quantization: NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity while higher-precision FP32 accumulation protects dot-product accuracy.

Ready for commercial and non-commercial use under OpenMDW-1.1 (included as LICENSE.md), consistent with Poolside's Acceptable Use Policy.

Model Summary

Evaluation Results

Scores are from the Laguna S 2.1 model card.

Quantization Details

• Quantization format: NVFP4 (compressed-tensors nvfp4-pack-quantized, group size 16) on routed-expert gate_proj / up_proj / down_proj; Poolside's official NVFP4 release, calibrated at the 1M-context configuration
• KV cache: FP8

Usage

Deploy with SGLang

# Poolside notes NVFP4 does not currently run correctly on SGLang; use vLLM below.

Deploy with vLLM

vllm serve AxionML/Laguna-S-2.1-NVFP4 \
    --enable-auto-tool-choice \
    --tool-call-parser poolside_v1 \
    --reasoning-parser poolside_v1 \
    --max-model-len 262144

Requires vLLM ≥ 0.25.0; quantization is detected automatically. Optional DFlash speculative decoding: add --speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":7,"method":"dflash"}' --max-num-seqs 32. Serve with the sampling defaults in generation_config.json.

Limitations

The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations. It may generate inaccurate, biased, or offensive content. Please refer to the original model card and the upstream quantized model card for full details.

Credits

• Base model: poolside/Laguna-S-2.1
• Quantization: poolside/Laguna-S-2.1-NVFP4 by Poolside
• Mirror: AxionML