Flint — FP8 routed experts, BF16 elsewhere

41.730 GB of weight tensors, 47.56% smaller than the 79.571 GB BF16 source. All routed-expert gate/up/down matrices in layers 2–48 use FP8 E4M3. Every other weight stays byte-identical to the original BF16: routers, shared experts, attention, sinks, norms, embeddings, output head and the two initial dense layers. There are 36,096 FP8 matrices and 638 unchanged source tensors. No NVFP4 tensors, activation quantization, training, or runtime correction are used in this checkpoint.

This is the practical FP8 candidate from the Flint quantization investigation. On 4,756 fresh multiple-choice questions its aggregate accuracy is close to BF16; its 93.89% next-token top-1 agreement still shows measurable numerical drift. General capability parity, including coding and reasoning, has not been established. The earlier 76.35 GB late-layer FP8 checkpoint is retained as a numerical control, not the recommended compression choice.

Compatibility: inference requires support for Flint's architecture and the block-scaled, weight-only FP8 tensor layout documented below. This release does not claim drop-in compatibility with a particular inference framework. The included correctness reference expands FP8 weights to BF16 and needs roughly 80 GB of GPU weight memory. Runtimes that consume compressed FP8 weights can avoid retaining that full BF16 expansion, but native compressed inference was not validated in this release. The 41.73 GB figure is tensor storage; runtime memory also includes activations, KV cache and workspace.

Source: RecursiveSI/flint at 2a247be0c0aec895ccb0ac7ca92229ec6df37288. Produced and checked on an RTX PRO 6000 Blackwell Server Edition (SM120).

Fresh task comparison

The candidates were frozen before scoring 660 ARC-Challenge test questions and 4,096 HellaSwag validation questions, excluding all questions from the previous 512-per-task subsets. This scores 19,023 continuations / 1,343,508 input tokens. The custom runner uses zero-shot, untemplated continuation likelihood and actual FlashInfer CUDA fused-add RMSNorm. Source and quantized paths use matching operation streams, attention buckets and MLP chunks. This is not an official evaluation-harness run.

FP8-minus-BF16 paired-bootstrap 95% intervals are [-1.21, +1.82] percentage points for ARC and [-0.02, +0.95] for HellaSwag. No statistical-equivalence margin was specified; these results do not prove parity or an improvement.

acc_norm divides continuation log-likelihood by character count. Raw accuracy is also reported: ARC 47.12% → 47.42% and HellaSwag 57.30% → 56.93% for BF16 → FP8. Normalized answer agreement is 92.88% / 96.61% for ARC / HellaSwag. FP8 changes 12 correct ARC answers to wrong and 14 wrong answers to correct; corresponding HellaSwag counts are 44 and 63. Similar aggregate scores can therefore hide changed answers.

Dataset revisions, exact token fixtures, every choice score, both scoring conventions, changed-answer counts and 10,000-draw bootstrap intervals are bundled in diagnostics/task-validation.json and reproduction/task-validation-corpus.json. The run had zero process swap. No serving throughput is reported.

Numerical fidelity and generation

A separate fresh corpus contains 49,140 next-token targets from 12 × 4,096-token sequences, balanced across prose, code and math text. All earlier test source rows were excluded. MBPP/GSM8K are text sources here, not official coding/math task evaluations. The paired paths use identical tokens and actual fused normalization.

Top-1 agreement measures numerical fidelity, not accuracy against labels. Small early perturbations can cross hard top-k routing boundaries and change later selected experts. Keeping router weights in BF16 does not keep router inputs or selected experts unchanged. A controlled FP32 experiment quantizing only layer 2 gives 95.54% agreement; supplying source expert IDs at later layers raises it to 99.85%. Matching hidden-state noise causes similar drift without quantized weights. Frozen teacher routes are a diagnostic, not a usable inference remedy. Low active parameter count has not been isolated as the cause.

The initial Python reference had a residual-normalization rounding mismatch; that was corrected for the fresh results above. The author's independent llama.cpp Flint branch also reproduces the FP8 drift: 93.06% agreement / KL 0.10559 on the smaller 6,138-target selection fixture with exactly decoded FP8 weights. This validates weight quality in a second runtime, not native FP8 kernels. Full SGLang runtime parity is not claimed.

Per-layer/expert retention, finer FP8, GPTQ and calibration-recovery prototypes did not establish a substantially better practical size/fidelity compromise. The 76.35 GB policy achieved 99.03% agreement but saved only 4.05%; it is a retention control. The complete evidence, rejected variants and limitations are in diagnostics/investigation.md.

Four fresh greedy generation smoke checks with the exported checkpoint and corrected normalization passed copying, arithmetic, a Python function and JSON, reaching EOS. generation-smoke.json contains complete transcripts and provenance. These are basic functionality checks, not coding-agent, reasoning or long-context evaluations.

Budgeted layer and footprint study

A follow-up checked every MoE layer in three contexts: introducing FP8 into BF16, restoring BF16 in the FP8 model, and introducing NVFP4 into the FP8 model. All 141 routed projection groups also received matched-input checks. Two disjoint text panels gave layer-rank correlations of 0.994, 0.145 and 0.399, respectively. The broad early/late effect reproduces; the exact ranking of where extra BF16 precision helps is much less stable.

The study then evaluated 72 complete policies under 28–48 GB footprint budgets, including equal-size depth, random, projection and local-swap comparisons. Finalists were frozen before a second, disjoint 49,140-target text holdout and 1,524 new task questions: 500 OpenBookQA test questions without supporting facts, and 1,024 previously unused HellaSwag questions. All use the same corrected normalization path within each paired comparison. These are additional fixtures, so their absolute scores differ from the earlier tables.

Selected reference points from this follow-up:

The full report gives every finalist's NVFP4/FP8/BF16 parameter and byte shares, raw and normalized accuracy, paired uncertainty intervals, numerical fidelity, precise layer/projection policies and same-size controls. Small score differences are not evidence of an accuracy improvement or equivalence. Candidate selection used text fidelity; this is an observed tradeoff study, not proof of a globally optimal accuracy frontier. The additional policy rows are experiments; this repository still contains the all-routed-expert FP8 weights described above.

Published follow-up: the 35.738 GB budget36-swap1 policy is now available as Flint FP8/NVFP4. Its exported tensors match the evaluated candidates, the serialized checkpoint exactly replays the text holdout, and four generation smoke checks completed. It saves another 14.36% relative to this FP8 checkpoint, with the lower numerical fidelity and uncertain task differences reported above. This repository retains its original all-expert FP8 weights.

Full layer and footprint report · Machine-readable results · Exact policies

FP8 tensor contract and integrity

Weights use E4M3 [N,K] payloads with one FP32 scale for each 128 × 128 block. The scale is absmax(block) / 448; zero blocks use 1. Payloads are row-major, without rotation or permutation. Scales are data-free and no labels are used for quantization.

The tensor suffixes are weight and weight_scale_inv ([N/128,K/128]). Despite its conventional name, the latter is a multiplicative dequantization scale:

BF16(float32(weight[n,k]) * weight_scale_inv[n//128,k//128])

precision-policy.json describes every routed projection. quantization-manifest.json pins source, producer code, precision counts and all 48 shard SHA-256 hashes. audit.json records the full tensor census, shapes/dtypes, exact equality to the evaluated FP8 candidate payloads, and byte preservation of all non-expert weights and tokenizer/template/generation/license files.

Reproduction

Use Python 3.12, CUDA PyTorch and FlashInfer with a supported CUDA toolkit. The measured environment used Torch 2.8.0+cu128, Triton 3.4.0, FlashInfer 0.6.16.post1 and CUDA 12.9 for norm compilation. Set CUDA_HOME to the toolkit; dependency pins and producer history are bundled. Generation requires enough VRAM for BF16-expanded weights.

hf download HawkBearPig/flint-FP8-BF16 --local-dir checkpoint
hf download RecursiveSI/flint --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --local-dir source
pip install -r checkpoint/reproduction/requirements.txt
export FLASHINFER_USE_CUDA_NORM=1
python checkpoint/reproduction/flint_fused_generate.py --checkpoint-dir checkpoint --prompts-json checkpoint/reproduction/smoke-prompts.json --output smoke.json
python checkpoint/reproduction/flint_quantize.py prepare --source source --work work --corpus checkpoint/reproduction/corpus-policy-validation.json --precision-policy checkpoint/precision-policy.json
python checkpoint/reproduction/flint_quantize.py export --source source --work work --corpus checkpoint/reproduction/corpus-policy-validation.json --precision-policy checkpoint/precision-policy.json --output reproduced
python checkpoint/reproduction/flint_quantize.py audit --source source --work work --corpus checkpoint/reproduction/corpus-policy-validation.json --output reproduced
python checkpoint/reproduction/flint_task_validation.py evaluate --source source --work work --corpus checkpoint/reproduction/task-validation-corpus.json --recipes fp8

The fresh task command reproduces source/FP8 scoring using the included frozen questions. The historical result also includes the mixed NVFP4/FP8 candidate, which requires its additional prepared NVFP4 layer files. Numerical details and hashes are under diagnostics; reproduction/file-hashes.json pins the bundled code.

The more compressed 30.80 GB NVFP4/FP8 candidate remains available with greater output-distribution drift. Source model and derivative are Apache 2.0; original architecture and weights are by Recursive.