flint-NVFP4-FP8

An experimental, weight-only quantization of Recursive Flint, produced on an RTX PRO 6000 Blackwell with 96 GB VRAM. Weight tensor payload: 30.80 GB (28.69 GiB), versus 79.57 GB for the BF16 source.

Accuracy parity has not been established. Quantization measurably changes the output distribution. This checkpoint is published for development and reproducible evaluation, not as a claim of negligible degradation. See the numerical investigation below.

Routed experts in layers 2–17 use FP8; those in layers 18–48 use NVFP4. Everything else is the source BF16 bytes, including attention, learned attention sinks, routers, shared experts, the first two dense MLPs, norms, embeddings, and the output head. Activations and KV remain BF16 in the quality reference; this release does not introduce activation quantization.

Source: RecursiveSI/flint at 2a247be0c0aec895ccb0ac7ca92229ec6df37288. A subsequent per-layer FP8/BF16 checkpoint keeps layers 0–44 BF16 and quantizes only routed experts in layers 45–48. It uses 76.35 GB and reaches 99.03% agreement / KL 0.000793 on fresh validation text, versus the much stronger compression and lower fidelity of this checkpoint. The all-expert FP8 candidate discussed below remains unpublished. These policies should not be confused.

Subsequent per-layer investigation

The early-layer sensitivity reproduces in the author's independent llama.cpp branch: FP8 only at layer 2 gives 95.21% agreement, versus 99.63% when only layer 48 is quantized. In an FP32 diagnostic, holding all later teacher expert selections fixed raises the first-layer FP8 result from 95.54% to 99.85%. Matched random hidden-state noise reproduces similar drift with no weight quantization. This identifies hard top-k routing as an amplifier of early perturbations; it does not establish that low active parameter count causes the sensitivity or that every early layer independently needs BF16.

A complete single-layer screen and joint layer/projection policy search followed, using actual FlashInfer CUDA fused normalization. The final primary policy was frozen before 49,140 fresh validation targets; exact layer results and comparisons are in diagnostics/layer-validation.json. The high-fidelity per-layer checkpoint above bundles its precise policy and reproduction code. The weights in this repository are unchanged; the update adds research and corrects the earlier link to an unpublished FP8 candidate.

Runtime status

This is a mixed-precision safetensors checkpoint for development of a native dgpp Flint port. Flint is not yet supported by dgpp, and this is not a drop-in Transformers, vLLM, SGLang, or ModelOpt export. The upstream model has custom architecture requirements beyond its weight format. The explicit dgpp_mixed config and tensor suffixes describe the intended loader contract.

A tested PyTorch reference is included under reproduction/. It expands quantized weights to BF16 for computation, so generation needs roughly 80 GB of GPU weight memory, regardless of this checkpoint's on-disk size. It is a correctness/quality reference, not an optimized low-memory inference engine. It does not claim bit parity with an upstream runtime.

hf download HawkBearPig/flint-NVFP4-FP8 --local-dir checkpoint
# Use a CUDA PyTorch environment; the recorded environment used torch 2.8.0+cu128.
pip install -r checkpoint/reproduction/requirements.txt
python checkpoint/reproduction/flint_reference.py --checkpoint-dir checkpoint \
  --prompt 'What is 17 + 25? Answer with the number only.' --max-new-tokens 128

Quantization recipe

• NVFP4: E2M1 weights, one E4M3 block scale per 16 consecutive input coordinates, and one positive FP32 reciprocal global scale per matrix. Two weights per byte; low nibble is the even coordinate. The global scale is 2688 / absmax(weight). A zero matrix uses global scale 1.
• NVFP4 scale optimization: test 31 factors from 0.70 through 1.30 (step 0.02) around the block's absmax scale, round each candidate to E4M3, and minimize squared weight reconstruction error. Codes use round-to-nearest-even on the E2M1 grid. No rotation, residual tensor, or runtime correction.
• FP8: E4M3 weights with FP32 absmax(block) / 448 scales on 128 × 128 blocks. Zero blocks use scale 1. Payloads are row-major, with no permutation or hidden transform.
• The shipped scale recipes do not use calibration activations. An activation-weighted alternative was screened and rejected. The precision assignment was chosen using separate selection samples.

NVFP4 tensors are weight_packed (U8 [N,K/2]), weight_scale (E4M3 [N,K/16]), and weight_global_scale (FP32 [1]). Decode as E2M1(code) * (float32(weight_scale) / weight_global_scale).

FP8 tensors are weight (E4M3 [N,K]) and weight_scale_inv (FP32 [N/128,K/128]). Despite the conventional suffix, weight_scale_inv stores a multiplicative dequantization scale: float32(weight[n,k]) * weight_scale_inv[n//128,k//128].

Quality measurements

The comparison uses identical operations and input token IDs within the included reference, BF16 matmuls and activations, FP32 attention scores/softmax with learned sinks, and an FP32 output head. The source checkpoint passed the Hugging Face checksum audit. All measured processes had zero swap. Six numerical tests cover codebook ties, zero blocks, scale optimization, FP8 block geometry, attention sinks/window/cache semantics, RoPE layout, and unbiased sigmoid-routing coefficients.

Initial held-out text test: 12 sequences of 4,096 tokens; 49,140 scored next tokens, balanced across WikiText-103 test prose, MBPP test code, and GSM8K test question/answer text. These are teacher-forced text samples, not official MBPP or GSM8K task-accuracy evaluations. Recipe selection used separate 1,024-token samples (6,138 scored tokens). Final test sequences were not used to fit scales or choose the layer boundary. Further numerical and task diagnostics were added after inspecting these results; those follow-ups are exploratory. Dataset revisions, row IDs, and token hashes are in evaluation.json.

This checkpoint by domain:

Top-1 agreement measures fidelity to BF16; it is not task accuracy. Small NLL improvements from quantization do not establish better model capability. These samples do not establish coding-agent benchmark parity, long-context capability at 262K, native-kernel numerics, or serving throughput. Four short greedy generation smoke tests (copying, arithmetic, Python, JSON) are included in generation-smoke.json; they are a basic functionality check only. For normal use preserve the upstream chat template and generation defaults (temperature 1.0, top-p 0.95, top-k 20), and its reasoning/tool-call conventions.

Numerical investigation and limitations

Flint selects six of 256 experts in each of its 47 MoE layers. Small changes to hidden states can change the selected experts even though router weights are unchanged. In an offline intervention, giving the FP8 path the BF16 teacher's expert IDs improves selection agreement from 92.65% to 96.89% and lowers KL from 0.09795 to 0.01859. This is evidence of routing amplification within the reference; frozen teacher routes are not deployable. Low active parameter count has not been isolated as a cause.

The original reference also has a residual/RMSNorm rounding mismatch with the released SGLang implementation. Replacing normalization with actual FlashInfer 0.6.16.post1 CUDA kernels yields 93.17% FP8 agreement and 89.65% mixed agreement on the selection set. Thus this mismatch matters, but does not explain away quantization drift. An independent paired run on the model author's llama.cpp Flint branch also finds 93.06% FP8 agreement, KL 0.10559, and NLL 2.03957 → 2.04378 using exactly decoded FP8 weights and the same 6,138 targets. This is independent weight-quality evidence, not a compressed-kernel test.

Even source BF16 weights are sensitive to execution: identical execution repeats exactly, but changing expert GEMM batching gives 94.77% agreement. This disagreement cannot be subtracted from quantization disagreement. An all-FP32 control is stable across batching, yet FP8 weights still yield 93.08% agreement in it. No full SGLang runtime parity is claimed.

On the original text test, FP8 agrees on 99.46% of the 30,022 positions with source logit margin at least 2; the mixed candidate agrees on 99.09%. Many flips involve close alternatives, but confident errors and distribution tails remain. These diagnostics do not replace the unfiltered agreement scores above.

Paired zero-shot, untemplated continuation scoring on 512 random questions per task:

The FP8-minus-BF16 paired bootstrap 95% intervals are [-2.34, +2.34] percentage points for ARC and [-1.17, +1.56] for HellaSwag. These subsets do not establish parity or an improvement, and equal aggregate scores can hide changed answers. They do not test chat, reasoning, coding agents, or long context. acc_norm uses continuation character length, following the task formatting, with a custom runner rather than an official harness run.

Finer FP8 blocks and fixed-scale GPTQ with 98,304 calibration tokens did not establish a clear FP8 improvement. Keeping the first 32 MoE layers BF16 gives 97.61% selection agreement with FP8 in the remaining 15 layers, but uses 67.49 GB. Keeping the first 44 MoE layers BF16 reaches 99.14% with the last three FP8, at 77.16 GB. These are exploratory retention controls, not additional releases. See diagnostics/investigation.md and diagnostics/ for the results, code, and provenance.

Follow-up: why retaining early BF16 layers helps

A controlled test quantizes only one layer, leaving every other weight unchanged. In FP32 arithmetic on the same 6,138 targets:

The initial hidden-state error is 0.84% relative RMS and grows to 12.81% by the last layer when only the first MoE layer is quantized. With expert selections fixed, it falls to 0.21%. The random-noise control matches each token's initial error norm. This shows that an early perturbation can trigger a cascade of changed routes; it is not evidence that every retained layer independently requires BF16. Keeping a long BF16 prefix prevents early perturbations and leaves fewer downstream routing decisions after quantization.

This is also not simply BF16 router rounding: using FP32 router calculations or stable tie breaking did not resolve the FP8 disagreement. Symmetric INT8 with 32-element row groups reduced weight error roughly fivefold, yet agreement was only 94.46%. Quantizing only the gate, up, or down expert projections yielded 93.29%, 93.70%, and 94.00%. The frozen-route diagnostic requires a teacher and is not a practical inference remedy. These results identify a mechanism within Flint, not a controlled comparison with other MoE architectures or proof that low active parameter count is the cause.

Integrity and reproduction

audit.json records a complete tensor census, shard SHA-256 verification, byte-for-byte equality of all 638 non-expert tensors and tokenizer/template/license files with the source, and exact equality of exported quantized tensors with the evaluated candidates. quantization-manifest.json records source revision, precision counts, shard sizes/hashes, package versions and scripts used for export. reproduction/file-hashes.json pins the bundled code; optional generation CLI fixes after export are recorded by these separate hashes.

Recreate the experiment with the bundled tools:

hf download RecursiveSI/flint --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --local-dir source
hf cache verify RecursiveSI/flint --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --local-dir source
python reproduction/flint_quantize.py corpus --source source --work work --corpus corpus.json
python reproduction/flint_quantize.py screen --source source --work work --corpus corpus.json
python reproduction/flint_quantize.py sensitivity --recipe mse --source source --work work --corpus corpus.json
python reproduction/flint_quantize.py fp8 --source source --work work --corpus corpus.json
python reproduction/flint_quantize.py corpus --source source --work work --corpus final.json --test-length 4096
python reproduction/flint_quantize.py evaluate --split test --recipe mse-fp8-2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17 --source source --work work --corpus final.json
python reproduction/flint_quantize.py export --recipe mse-fp8-2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17 --source source --work work --corpus final.json --output checkpoint
python reproduction/flint_quantize.py audit --source source --work work --corpus final.json --output checkpoint
python reproduction/flint_quantization_test.py

The source model and this derivative are released under Apache 2.0. Architecture and original weights are by Recursive. Quantization and evaluation tooling is from dgpp. This release does not change the source model's stated capabilities, context configuration, or limitations.