Sage S2.1 Flash for two GB10s: 4-bit routed experts, the official FP8 release's attention and shared experts, BF16 where the release keeps BF16.
| At a glance | |
|---|---|
| 🎯 Closeness to BF16 | KL 0.0426, top-1 agreement 93.56%, NLL 1.05% above BF16's, in PyTorch on 128 documents |
| 🔢 What is 4-bit | The routed experts of layers 3 to 44 and of the MTP layer: NVFP4 (E2M1 codes, an E4M3 scale every 16 weights), quantized here from BF16 with no calibration data |
| 🧩 What is not | The official FP8 release's bytes for MLA attention and the shared experts; MXFP8 for the dense MLPs of layers 0 to 2; BF16 for KDA attention, lm_head, embeddings and the vision tower, as the FP8 release keeps them |
| 💾 Size | 183.8 GB in 144 files, 133 of them safetensors (the BF16 checkpoint is 643 GB, the official FP8 release 328.5 GB) |
| ⚡ Serving | TP=2 on two GB10s: 92.51 GiB of model on the first machine, 19.9 tok/s with drafts off, 6.7 s to the first token of a cold 9,600-token prompt |
| 🔧 Engine | TensorFold release 1.2.9, built from its tag, and main at 077998f7d9 both serve it (The engine). No engine change is needed: every form here is one release 1.2.9 reads |
TP=2 means the model split across two systems.
Sage-S2.1-Flash Mixed NVFP4 is a derived checkpoint of internlm/Sage-S2.1-Flash for GB10 (DGX Spark class) systems. Sage S2.1 Flash has the shape of GLM-5.3-Flash: 45 layers and an MTP draft layer, with 288 routed experts a layer from layer 3 on. The routed experts of layers 3 to 44 hold 94.7% of the BF16 weights. This checkpoint stores the routed experts in 4 bits and leaves the rest at the official FP8 release's precision, except the three dense MLPs, so that the model fits two GB10s. It is not a Shanghai AI Laboratory, InternLM, Z.AI or NVIDIA release.
It is the plain starting point, made from the weights alone. A second checkpoint is planned in the form of GLM-5.3-Flash Jammed XSFP4 NZ5, with scales every 32 weights and corrections learned from calibration text; it is not made yet, and nothing of it is in this one.
csf_32x2_v1, and the mix of forms is declared in ModelOpt's MIXED_PRECISION table; TensorFold reads both, and no other loader was tried. The source's deployment_guide.md covers LMDeploy, vLLM and SGLang with the official releases.projectors/ folder is not in this checkpoint.| Component | Stored precision | How it was made |
|---|---|---|
| Decoder routed experts, layers 3 to 44 163.02 GB |
NVFP4: 4-bit E2M1 codes; E4M3 block scales of 16, stored as csf_32x2_v1; FP32 global scale, a power of two; 36,288 matrices |
Quantized here from BF16. Calibration: none. Smoothing: none. Scales: for each block of 16 weights, the E4M3 scale with the smallest squared error among 25 neighbouring values, from 10 steps below to 14 above the first one that reaches the block's largest weight divided by 6. No activation statistics. |
| MTP routed experts, layer 45 4.08 GB |
NVFP4 as above, block scales stored directly; 864 matrices | The same fit. |
MLA attention (q_a, q_b, kv_a, o) in 11 layers and the MTP layer, and the shared experts of layers 3 to 452.29 GB |
FP8 E4M3 in 128×128 blocks, an FP32 scale a block; 177 matrices | The official FP8 release's bytes, revision 716e6436905650b7be93e1b7dedfb0bb95afd3a3. Its scale tensor weight_scale_inv is stored here as weight_scale; the value multiplies the code in both. |
| Dense MLP, layers 0 to 2 0.47 GB |
MXFP8: E4M3 values, a UE8M0 scale every 32 weights; 9 matrices | Made here from BF16: each block's scale is the smallest power of two that keeps its largest weight within E4M3's range. The official release stores these in 128×128 FP8 blocks; TensorFold's loader at 077998f7d9 takes NVFP4 or MXFP8 for these three layers, so they are remade. |
KDA attention (q, k, v, o) in 34 layers, lm_head, embeddings, router, kv_b, indexers, norms, the vision tower13.86 GB |
BF16; F32 where the source has F32 | The BF16 checkpoint's bytes, revision 76f4513db7464485605a002a06a6190de0460574. |
The FP8 bytes taken here are the BF16 weights rounded on the release's own scales: rounding this BF16 checkpoint on those scales gives the 177 matrices' codes byte for byte. So the 8-bit parts here are a plain rounding of BF16 too, with no retraining in them. The rest of the official release (its routed experts and dense MLPs) was not compared.
No engine change is needed. TensorFold's release 1.2.9 reads every form stored here, by the config's MIXED_PRECISION table.
| Engine | Serves this checkpoint? |
|---|---|
TensorFold 1.2.9, the latest release as of 2026-11-17, built here from its tag (6b9de69510) |
Yes, in one launch at TP=2 on two GB10s, with a 40,960-token window, one stream and vision off. The three fixed prompts of In serving gave the same token_sha256 as on the build below, with drafts and with "draft": false. The model took 92.51 GiB on the first machine and loaded in 115 s; decode with drafts off ran at 19.95 and 19.91 tok/s; free memory at its lowest was 8.62 and 8.67 GiB on the two machines. Cold prompts and image input were not run on this build |
TensorFold main at 077998f7d9, 186 commits after 1.2.9 |
Yes. Everything under In serving was measured on this build, the image check included |
| Earlier TensorFold releases | Not tried |
| Other loaders | Not tried (Is it for you?) |
To serve it, put this repository's files in one folder on each machine and follow TensorFold's GLM-5.3-Flash CUDA recipe, which places the ranks and names the link between the hosts for NCCL. The launches here named that folder to the engine's binary, the same line on both machines but for the rank, with 192.0.2.1 standing for the first machine's address on their direct link:
# on the second machine (rank 1); on the first, TF_GLM_RANK=0
TF_GLM_RANK=1 TF_GLM_WORLD=2 NCCL_COMM_ID=192.0.2.1:29583 \
TENSORFOLD_MEMORY_RESERVE_GIB=9 TF_KV_SHARE=16 TENSORFOLD_PREFILL_SHARE=0.7 \
tensorfold-native serve /path/to/Sage-S2.1-Flash-Mixed-NVFP4 \
--context 40960 --parallel 1 --max-tokens 32768 --reasoning-effort high
That line is the configuration tried; beside it the launches set a name, host and port for the server, --no-update-check (a no-op in the engine), the engine's heat limits (TF_HEAT_HIGH=92, TF_HEAT_LOW=88) and NCCL's settings for the link. It was not weighed against others. The image check added --vision and kept 4 prompt states (TF_KV_SHARE=4). Memory is tight with these settings (In serving): with image input at a 262,144-token window the 9 GiB reserve was too small and 11 GiB served, and the engine's default reserve (a tenth of RAM) and more than one stream were not tried. The folder served held the 144 files of SHA256SUMS. tensorfold pull and serving by the repository's name were not tried.
128 documents (32 each of code, English, Japanese and math; 144,285 scored positions) went through the model in PyTorch twice: once with the BF16 weights and once with this checkpoint's weights dequantized, activations in 16 bits both times. Every number below compares the two runs' next-token distributions over the whole vocabulary at the same positions.
| Documents | Positions | KL (BF16 ‖ this) | Top-1 agreement | NLL, BF16 → this |
|---|---|---|---|---|
| All 128 | 144,285 | 0.0426 (0.0384 to 0.0471) | 93.56% (93.09 to 94.05) | 1.0385 → 1.0494 (+1.05%) |
| Code | 30,966 | 0.0315 | 96.40% | 0.4846 → 0.4957 (+2.29%) |
| English | 47,458 | 0.0534 | 92.81% | 0.9576 → 0.9774 (+2.07%) |
| Japanese | 58,785 | 0.0411 | 92.53% | 1.4284 → 1.4332 (+0.34%) |
| Math | 7,076 | 0.0306 | 94.80% | 0.7655 → 0.7671 (+0.20%) |
The ranges are 95% intervals from resampling whole documents. The experts the router picks for a token overlap BF16's picks with a Jaccard index of 0.844 on average over the 42 routed layers: 0.961 at layer 3, 0.771 at layer 37, the lowest.
These are PyTorch numbers, not the serving engine's. The document set and its token ids are the ones behind the GLM-5.3-Flash cards of this account (Sage's tokenizer.json is the same file), but the model is another, so the numbers do not line up with those cards'.
The MTP layer is not part of the numbers above: the main model does not run it. Measured apart, on the same documents, with the document's own tokens fed in and one draft deep, its draft of the token after next agrees with the main model's own choice at 77.45% of 144,157 positions with this checkpoint's layer, against 77.29% with BF16's layer on BF16's run; the difference, +0.16 points, has a 95% interval of −0.52 to +0.83 over documents.
Two GB10s at TP=2 over a direct link, TensorFold main at 077998f7d9, a 40,960-token window, one stream. Two launches with vision off: the first for the token check and decode, the second for the prompts. A third launch with vision on is below the table.
| Model on the first machine | 92.51 GiB; 116 s and 114 s to load |
Free memory at its lowest (MemAvailable) |
8.65 to 8.73 GiB on each machine, over both launches |
| Decode, MTP drafts off | 19.94 and 19.91 tok/s (two prompts, 256 tokens five times each, medians) |
| With drafts on (the MTP layer and copies of earlier text) | 5.3, 2.0 and 3.8 tokens a round on the three fixed prompts (counting, prose, code); the log counts 415 of 648, 250 of 505 and 377 of 641 drafts accepted |
| Cold prompt of 9,600 tokens | 6.69 s to the first token |
| Cold prompt of 38,400 tokens | 26.15 s to the first token |
Drafted against "draft": false |
The same 512 tokens on each of the three fixed prompts (token_sha256 82b9fd2bc80a…, 3d3f24d81246…, 1ee72b9053a2…) |
TensorFold times two forms of the experts at load and keeps the faster: the first launch kept factored, the second classic. No line of either launch's logs says disabled, fallback or compile failed.
Memory is the tight part. Free memory fell to 8.65 GiB on each machine, 0.65 GiB above the 8 GiB at which this setup's watchdog stops the engine. The engine's own token pool was not the limit: after the model and its 9 GiB reserve it had 15.7 GiB, room for 23 windows of 40,960 tokens by its own count. The longest prompt of these launches was 38,400 tokens; a 262,144-token window is below, and more than one stream was not tried.
Image input, one check. A third launch with vision on loads the vision tower on the first machine: 93.78 GiB of model there. To make room it kept 4 prompt states instead of the 16 of the launches above (0.9 GiB against 3.6 GiB); with 16 it was not tried. Two 448×448 PNGs made for the check, a red square and a blue square on white, were each sent with the question "What color is the square?": the replies were "Red." and "Blue.", the same with drafts and with "draft": false, at 282 prompt tokens a request. Free memory at its lowest was 8.72 GiB on the first machine and 10.26 GiB on the second.
A 262,144-token window, tool calls and larger images. Two later launches on another pair of GB10s, the same build, with a 262,144-token window, vision on, 4 kept prompt states and one stream; the model takes 93.90 GiB on the first machine.
| With the 9 GiB reserve of the launches above | It loads, with 8.55 GiB free, and gives the three fixed prompts' hashes. The first image request took free memory under 8 GiB, and the watchdog stopped the engine |
| With an 11 GiB reserve | 10.64 GiB free once loaded; the engine counts room for 3 windows of 262,144 tokens |
| The three fixed prompts | The same token_sha256 as above, drafted and with "draft": false |
| Tool calls | A round trip in 5 cases, each with reasoning_effort high and without: a call with JSON arguments (finish_reason tool_calls), an answer from the tool's result, one tool picked out of three, the same call put together from a stream's deltas, and a plain answer where no tool is needed. 10 of 10 as expected |
| Images | Squares of 448, 896, 1,344 and 2,016 pixels a side (282 to 4,122 prompt tokens), each answered with its colour |
| A cold prompt of 200,000 tokens | 137.9 s to the first token; sent again, 1.1 s, with 198,656 tokens from the prompt cache |
| Free memory at its lowest | 9.63 GiB on the first machine, during the 2,016-pixel image; the long prompt did not lower it |
Not run: TP=3 or wider, more than one stream, video, the scientific-input projector, and the control-token handling the source's deployment_guide.md asks for.
The fit. Each routed expert has three matrices: gate, up and down. Each is cut into blocks of 16 weights along a row. A block is stored as 16 E2M1 codes and one E4M3 scale; a weight reads as code × scale × the matrix's global scale. For each block the scale is the one, among 25 E4M3 values around the block's largest weight divided by 6, that leaves the smallest squared error after rounding. In the 42 routed layers and the MTP layer, no block's best scale fell at either end of the search.
The global scale is a power of two, one for an expert's gate and up and one for its down, set so that the search above never reaches E4M3's largest value.
The packing. A row's block scales are stored as csf_32x2_v1, the lossless form nvidia/GLM-5.3-Flash-FP4-Local uses: a base byte a row, a 4-bit step a scale, and a list of the scales that fall outside the row's 16 steps (1.48% of the 19.0 billion scales of layers 3 to 44). Every packed scale was decoded again and compared with the fitted bytes.
Nothing learned. No calibration text, no activation statistics and no distillation went into this checkpoint.
| Part | Origin |
|---|---|
| BF16 weights | internlm/Sage-S2.1-Flash, revision 76f4513db7464485605a002a06a6190de0460574 |
| FP8 weights of MLA attention and the shared experts | internlm/Sage-S2.1-Flash-FP8, revision 716e6436905650b7be93e1b7dedfb0bb95afd3a3 |
| Architecture | GLM-5.3-Flash's (zai-org/GLM-5.3-Flash, MIT): the same tensor names and shapes |
| Scale packing | csf_32x2_v1, from local-inference-lab/flashinfer (Apache-2.0), as used by nvidia/GLM-5.3-Flash-FP4-Local |
| License | MIT, by the source README's statement; GLM-5.3-Flash's MIT text in LICENSE-GLM-5.3-Flash |
| Files | What they hold |
|---|---|
e2m1-{gate,up,down}-L03.safetensors to -L44 (126 files, 163.02 GB) |
The routed experts of one layer and one projection each: 288 experts' codes, packed scales, global scale and input_scale |
e2m1-{gate,up,down}-L45.safetensors (3 files, 4.08 GB) |
The MTP layer's routed experts, block scales stored directly |
model-shell-0000{1..4}-of-00004.safetensors (16.62 GB) |
Everything else: 13.86 GB of BF16 and F32, 2.29 GB of FP8, 0.47 GB of MXFP8 |
model.safetensors.index.json |
Which file holds each of the 295,564 tensors |
config.json, hf_quant_config.json |
The model's config with ModelOpt's MIXED_PRECISION table: an entry for each quantized matrix, and one for each layer's routed experts |
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, processor_config.json, processing_glm5.py, tokenization_glm5.py, deployment_guide.md |
Copied from the source repository at the revision above |
LICENSE-GLM-5.3-Flash |
GLM-5.3-Flash's MIT text |
SHA256SUMS |
The sha256 of the checkpoint's 144 files: every row above but the license |
Left out: the source's README.md and README_CN.md (linked above), assets/, projectors/ and vllm_patch/.
MIT. The source repository, internlm/Sage-S2.1-Flash, states in its README: "The code and model weights in this repository are released under the MIT License." The official FP8 repository, internlm/Sage-S2.1-Flash-FP8, whose bytes are 2.29 GB of this checkpoint, has the same sentence. As of 2026-11-17 neither ships a LICENSE file or has a licence field on its card, so that sentence is this checkpoint's basis, and there is no copyright line of Sage's to reproduce here. Sage S2.1 Flash has GLM-5.3-Flash's architecture; GLM-5.3-Flash is under the MIT License (Copyright (c) 2026 Z.AI Co., Ltd), whose text is in LICENSE-GLM-5.3-Flash. No calibration data was used.
model-shell-00003-of-00004.safetensors 5.4 GBmodel-shell-00002-of-00004.safetensors 5.4 GBmodel-shell-00001-of-00004.safetensors 5.3 GBe2m1-down-L45.safetensors 1.4 GBe2m1-gate-L45.safetensors 1.4 GBe2m1-up-L45.safetensors 1.4 GBe2m1-down-L44.safetensors 1.3 GBe2m1-down-L20.safetensors 1.3 GBe2m1-down-L13.safetensors 1.3 GBe2m1-down-L21.safetensors 1.3 GBe2m1-down-L17.safetensors 1.3 GBe2m1-down-L18.safetensors 1.3 GBe2m1-down-L43.safetensors 1.3 GBe2m1-down-L19.safetensors 1.3 GBe2m1-down-L16.safetensors 1.3 GBe2m1-down-L42.safetensors 1.3 GBe2m1-down-L24.safetensors 1.3 GBe2m1-down-L23.safetensors 1.3 GBe2m1-down-L12.safetensors 1.3 GBe2m1-down-L15.safetensors 1.3 GBe2m1-down-L14.safetensors 1.3 GBe2m1-down-L22.safetensors 1.3 GBe2m1-down-L41.safetensors 1.3 GBe2m1-down-L37.safetensors 1.3 GBe2m1-down-L40.safetensors 1.3 GBe2m1-down-L28.safetensors 1.3 GBe2m1-down-L26.safetensors 1.3 GBe2m1-down-L27.safetensors 1.3 GBe2m1-down-L25.safetensors 1.3 GBe2m1-down-L11.safetensors 1.3 GBe2m1-down-L38.safetensors 1.3 GBe2m1-down-L36.safetensors 1.3 GBe2m1-down-L29.safetensors 1.3 GBe2m1-down-L39.safetensors 1.3 GBe2m1-down-L30.safetensors 1.3 GBe2m1-down-L32.safetensors 1.3 GBe2m1-down-L35.safetensors 1.3 GBe2m1-down-L34.safetensors 1.3 GBe2m1-up-L12.safetensors 1.3 GBe2m1-up-L14.safetensors 1.3 GBe2m1-down-L08.safetensors 1.3 GBe2m1-down-L33.safetensors 1.3 GBe2m1-down-L07.safetensors 1.3 GBe2m1-down-L31.safetensors 1.3 GBe2m1-down-L09.safetensors 1.3 GBe2m1-down-L10.safetensors 1.3 GBe2m1-down-L04.safetensors 1.3 GBe2m1-up-L15.safetensors 1.3 GBe2m1-down-L06.safetensors 1.3 GBe2m1-down-L05.safetensors 1.3 GBe2m1-up-L16.safetensors 1.3 GBe2m1-up-L13.safetensors 1.3 GBe2m1-up-L19.safetensors 1.3 GBe2m1-down-L03.safetensors 1.3 GBe2m1-up-L17.safetensors 1.3 GBe2m1-gate-L12.safetensors 1.3 GBe2m1-gate-L15.safetensors 1.3 GBe2m1-up-L18.safetensors 1.3 GBe2m1-gate-L27.safetensors 1.3 GBe2m1-gate-L25.safetensors 1.3 GBe2m1-gate-L14.safetensors 1.3 GBe2m1-gate-L34.safetensors 1.3 GBe2m1-gate-L26.safetensors 1.3 GBe2m1-up-L28.safetensors 1.3 GBe2m1-up-L25.safetensors 1.3 GBe2m1-up-L27.safetensors 1.3 GBe2m1-gate-L35.safetensors 1.3 GBe2m1-gate-L28.safetensors 1.3 GBe2m1-gate-L16.safetensors 1.3 GBe2m1-gate-L36.safetensors 1.3 GBe2m1-gate-L30.safetensors 1.3 GBe2m1-gate-L33.safetensors 1.3 GBe2m1-up-L20.safetensors 1.3 GBe2m1-up-L26.safetensors 1.3 GBe2m1-gate-L31.safetensors 1.3 GBe2m1-up-L30.safetensors 1.3 GBe2m1-gate-L11.safetensors 1.3 GBe2m1-up-L11.safetensors 1.3 GBe2m1-gate-L39.safetensors 1.3 GBe2m1-gate-L37.safetensors 1.3 GBe2m1-up-L24.safetensors 1.3 GBe2m1-gate-L19.safetensors 1.3 GBe2m1-gate-L32.safetensors 1.3 GBe2m1-gate-L38.safetensors 1.3 GBe2m1-up-L08.safetensors 1.3 GBe2m1-gate-L13.safetensors 1.3 GBe2m1-gate-L09.safetensors 1.3 GBe2m1-up-L21.safetensors 1.3 GBe2m1-up-L09.safetensors 1.3 GBe2m1-gate-L08.safetensors 1.3 GBe2m1-up-L31.safetensors 1.3 GBe2m1-up-L07.safetensors 1.3 GBe2m1-gate-L07.safetensors 1.3 GBe2m1-gate-L04.safetensors 1.3 GBe2m1-up-L33.safetensors 1.3 GBe2m1-up-L22.safetensors 1.3 GBe2m1-up-L34.safetensors 1.3 GBe2m1-up-L04.safetensors 1.3 GBe2m1-gate-L29.safetensors 1.3 GBe2m1-gate-L10.safetensors 1.3 GBe2m1-up-L05.safetensors 1.3 GBe2m1-up-L37.safetensors 1.3 GBe2m1-gate-L05.safetensors 1.3 GBe2m1-up-L10.safetensors 1.3 GBe2m1-up-L06.safetensors 1.3 GBe2m1-up-L38.safetensors 1.3 GBe2m1-up-L36.safetensors 1.3 GBe2m1-up-L29.safetensors 1.3 GBe2m1-up-L35.safetensors 1.3 GBe2m1-up-L23.safetensors 1.3 GBe2m1-gate-L06.safetensors 1.3 GBe2m1-up-L32.safetensors 1.3 GBe2m1-up-L40.safetensors 1.3 GBe2m1-up-L39.safetensors 1.3 GBe2m1-gate-L17.safetensors 1.3 GBe2m1-gate-L40.safetensors 1.3 GBe2m1-gate-L18.safetensors 1.3 GBe2m1-up-L41.safetensors 1.3 GBe2m1-gate-L20.safetensors 1.3 GBe2m1-gate-L03.safetensors 1.3 GBe2m1-up-L43.safetensors 1.3 GBe2m1-up-L42.safetensors 1.3 GBe2m1-gate-L24.safetensors 1.3 GBe2m1-up-L03.safetensors 1.3 GBe2m1-gate-L22.safetensors 1.3 GBe2m1-gate-L21.safetensors 1.3 GBe2m1-gate-L41.safetensors 1.3 GBe2m1-gate-L23.safetensors 1.3 GBe2m1-up-L44.safetensors 1.3 GBe2m1-gate-L42.safetensors 1.3 GBe2m1-gate-L43.safetensors 1.3 GBe2m1-gate-L44.safetensors 1.3 GBmodel-shell-00004-of-00004.safetensors 548 MBinternlm/Sage-S2.1-Flash