Flint NVFP4
Converted from RecursiveSI/flint at revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 with MLX-VLM main, version 0.7.10, and MLX 0.32.4.
NVFP4 weight quantization using nvfp4 with group size 16. All 47 MoE routers use 8-bit affine quantization with group size 64. Normalization parameters, attention sinks and other non-quantized parameters remain BF16. Each linear has an explicit FP32 global scale (absmax/2688), independently per expert for expert projections. Embeddings use 8-bit affine/group-64. These scales prevent small block scales from underflowing; the stock unscaled NVFP4 conversion failed the smoke checks and is not published.
The upstream tokenizer, chat template, generation defaults and Apache-2.0 license are preserved byte for byte. Thinking remains controlled by the original template. No thinking-disable override is included. See the source model card for architecture, intended use and limitations. These converted weights have not been evaluated on the source model's published benchmark suite.
Usage
pip install -U git+https://github.com/Blaizzy/mlx-vlm.git@6d9ba328abbd6129c7461eda86280aa9fbfd9986
mlx_vlm.generate --model nativ-community/flint-nvfp4 --prompt "Write a Python function that merges overlapping intervals." --max-tokens 1024
The source defaults are temperature 1.0, top-p 0.95 and top-k 20. Reasoning consumes the output budget; use a larger budget for complex tasks. The source configuration supports a combined prompt/output window of 262,144 tokens; this conversion's smoke tests do not validate long-context quality.
Reproduce conversion
Download convert_nvfp4.py from this repository first. It uses the existing MLX-VLM scaled-NVFP4 modules; no runtime patch is required.
pip install mlx==0.32.4
python convert_nvfp4.py --hf-path RecursiveSI/flint --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --mlx-path flint-nvfp4
Use the pinned MLX-VLM commit and MLX version above. The conversion preserves model-specific router precision. Options, per-module overrides and all shard SHA-256 hashes are recorded in conversion.json. The original tokenizer, template, generation config and license are copied unchanged after conversion. SHA256SUMS covers the published artifacts.
Validation
The saved checkpoint passed strict reloading, finite prefill/decode logits, cache-position checks, arithmetic, short retrieval and simple Python-function smoke checks on Apple M5 Max with MLX 0.32.4. All three prompts used the native template, greedy decoding and a 512-token output budget. See validation.json for exact prompts, outputs and checks. Generated code was parsed as an AST, not executed.
These are smoke checks, not a quality benchmark or proof of equivalence to BF16 or Transformers. Quantization can change answers, and floating-point 4-bit formats may differ in quality from affine quantization.