Ling-3.1-flash VQ GGUF

Experimental VQ8 conversion of the official text-only Ling-3.1-flash checkpoint, pinned to a4ccd6b45c002ddb236f9cb77d52cc427482bf19.

This release uses a custom ling31_vq GGUF family. It requires a runtime that implements FORMAT.md; ordinary llama.cpp, Ollama and Transformers loaders do not implement this format. ds4-dfm-rs runs it on one DGX Spark (GB10): recipe v1 from 81eade8a (#92), recipe v2 with #95 (the long-context round), whose 709cf8df the measurements below use.

Selected recipe: ling31-vq8-mix8r13-60r12-mtp13-q4h-q8e-q4fg-v2. v2 replaces only body-nonexpert.gguf: the 126 KDA decay/gate projections are Q4_K instead of block-scaled FP8 and the embeddings Q8_0 instead of BF16 (trunk 116.4 → 114.0 GB). It was derived from v1's part with the converter's own quantizer; RECIPE.md and the derivation receipt record how. v1 (ling31-vq8-mix8r13-60r12-mtp13-q4h-v1) remains at revision e29aa7d8. Conversion and file-integrity checks are complete, and all 71 GGUF parts are published. The original-model baseline completed 13 held-out cases; the final A/B quality comparison and final selected MTP comparison were not completed. See QUALITY.md for the recorded scope and limitations. The assembly receipt records verified file sizes, hashes and complete source-tensor accounting. This publication does not claim a final model-quality pass.

Measured on DGX Spark (GB10)

Measured on this artifact (v2) with ds4-dfm-rs 709cf8df. One warm session per process, 2,048-token incremental prefill and 128 forced greedy tokens per frontier. Curves are per-frontier medians; bands are the observed min–max of three fresh runs per variant.

v1 measured 778.8 / 14.86 (MTP off) and 859.8 / 21.15 (two drafts) at a64e323b (#93).

• Decode gains 16% at 2K and 20% at 64K over v1: Q4_K decay/gate remove 1.85 GB of each token's 12.6 GB of weight reads, and ds4-dfm-rs now keeps MLA latent rows as FP8 (641 instead of 1,152 bytes per token and layer). Incremental prefill falls 35% from 4K (964.9) to 64K (625.8): the nine MLA layers expand each 2,048-key segment of the latent cache in every step.
• MTP drafts greedy requests only; thinking requests, which sample at temperature 1.0, decode without drafts. Two drafts measured best.
• ds4-server defaults to the native 262,144-token context: two persistent banks at 2,048-row prefill chunks with v2 (one with v1), or one bank with two MTP drafts (--mtp-mode on --mtp-draft 2). After a restart, the next turn of a 124K-token conversation restored 123,931 of 123,957 tokens from disk KV in 3.7 s.
• CUDA 13.4, sm_121a, 300–2,200 MHz clock policy; idle 299–305 MHz before each run and 2,184 MHz in one sample right after a run (not sampled during the sweep).

Model and precision

The original model has 72 trunk layers: 63 KDA and nine gated MLA blocks. Layers 0–3 use dense FFNs; layers 4–71 have 512 routed experts and select eight per token. Hidden width is 4096, expert intermediate width is 1280, and the vocabulary contains 157184 tokens. Layer 72 is a separate MLA/MoE MTP block. There is no vision encoder, Engram, PLE, mHC or sparse attention indexer in this checkpoint.

Routed expert weights use vectors of eight consecutive input-axis weights, one independently trained E4M3 codebook shared by all expert projections in each layer, and FP16 row RMS gains. The selected mixed allocation and its screening evidence are recorded in RECIPE.md and recipe.json. Trunk layers 4, 5, 6, 9, 12, 43, 57 and 58 use K8192; the other 60 routed trunk layers use K4096. MTP layer 72 uses a separate K8192 codebook. K4096 uses 12-bit indices (1.5 index bits/weight); K8192 uses 13-bit indices (1.625 index bits/weight). Gains, codebooks and container overhead are additional. The 13th bit is a separate bit plane. Each K4096 codebook was trained separately; it is not a truncated K8192 codebook.

The conservative nonexpert policy quantizes 504 large projections to Q4_K and uses a separately evaluated Q4_K output head. Recipe v2 also stores the 126 KDA decay/gate (f/g) projections as Q4_K, encoded from the FP8 values times their block scales rounded to BF16, and the embeddings as Q8_0. Router, bias, norms, b-projection, convolution, recurrence controls, MLA kv_b and head gates retain their original mixed precision. MTP nonexpert weights retain their original precision. No zchain, GR/RB correction, LoRA, QAT, pruning or distillation is applied. All quantized weights originate directly from the pinned original checkpoint.

Files and optional MTP

release.json is the loading entrypoint. These are independent GGUF parts, not standard numbered GGUF splits; do not concatenate them.

body-nonexpert.gguf contains the attention and dense FFN weights for layers 0–3, plus the other trunk layers' nonexpert weights, output head and embeddings.

The optional MTP experts use their own K8192 codebook and FP16 row gains. MTP is an approximately 8.17B-parameter block, not merely an extra output head. With MTP disabled, select the trunk feature before mapping or uploading parts; there must be no MTP GPU weights, repacking, cache or workspace allocation. The selected MTP parts contain no duplicate original expert weights.

tensor-manifest.json maps each of the 214025 original tensors exactly once to its preserved or quantized output and lists the generated codebooks. files.json and SHA256SUMS identify the released file contents. Configuration, tokenizer, chat template and special-token files are unchanged original assets. See SOURCE.md for provenance and licensing. The unchanged config.json describes the original FP8 checkpoint; the release recipe and tensor manifest define the stored GGUF precisions.

Numerical and quality scope

The numerical reference separates original-checkpoint inference (A) from inference using the exact stored quantized bytes (B). Original FP8 values use their multiplicative F32 block scales, then the declared BF16 weight boundary. VQ and Q4_K stored values are independently reconstructed before the same boundary. Router and recurrence controls follow the documented F32 contract; SwiGLU clamps follow the pinned serving implementations, including the distinct shared/routed layer lists and unclamped MTP layer.

Original-model baseline A completed all 13 held-out cases: 12 generations ended with normal EOS, and the Korean story reached its 768-token cap without a completed ending. The code answer passed six functional cases but included Markdown fences despite a code-only instruction. These are baseline findings; they do not measure the released VQ model's quality. Final matched A/B NLL/PPL, KL, top-1 agreement, generation-quality results and the final selected MTP comparison were not completed. Screening measurements are not a final model-quality certificate. QUALITY.md summarizes the evidence.

A runtime screen with the ds4-dfm-rs CUDA backend teacher-forced the same 13 held-out cases on reference A's token ids: NLL 1.043 (v1 1.046) against A's 1.117 over the same 642 targets, higher than A in 7 of 13 cases, with 86.8% top-1 agreement. The targets are fixture text, so neither model is favoured; 13 cases do not measure a quality gain, and this screen is not the final A/B. Against v1, v2 moved the 32K–192K frontier logits by KL 0.00003–0.0322 with the same argmax, the same order as switching between 1,024- and 2,048-row prefill chunks (0.0002–0.0815 at 32K–128K); teacher-forced NLL after 32K, 64K and 128K prompts was 2.071 against v1's 2.080. FP8 activation execution, gain-after-dot kernels and direct Q4_K serving arithmetic are outside this reference comparison. BF16 reconstruction in the quality reference does not establish serving speed or dequantization cost.

The original EOS is 156895 (<|role_end|>); padding is 156892. The original thinking and tool-call tokens and chat template are preserved. Greedy diagnostics are distinct from official sampling with temperature 1.0, top-p 0.95 and top-k 20.

Memory and context

The assembled 71 GGUF parts occupy 115760613632 bytes (107.810473 GiB), including MTP. The trunk accounts for 113960788096 bytes and optional MTP adds 1799825536 bytes (v1: 118214943744 bytes in all). These are verified written sizes, including container headers, scales, codebooks, gains and padding. Tokenizer, documentation and audit manifests are additional files.

memory-budget.json models v1's part sizes, batch-one KDA states and BF16 latent MLA caches. It compares conditional residency scenarios against a historical DGX Spark MemTotal of 130596163584 bytes (121.627155304 GiB), which has not been rechecked on the target machine. The budget reserves 4 GiB for OS/driver, 3 GiB for shared workspace and 1 GiB for loading.

These are budget calculations, not measured GB10 peaks or supported-context claims. Measured with ds4-dfm-rs, a 262,144-token session at 2,048-row chunks takes 3.12 GiB (1.41 GiB FP8 latent KV, 276 MiB KDA state, 186 MiB controls, 1.26 GiB workspace), each further bank 1.68 GiB, and the first requests add 0.41–0.48 GiB as CUDA loads kernel images and workspaces. Resident repacking, duplicate CPU weights and persistent BF16 expansion require additional memory. MTP draft/verify workspace must fit the shared allowance. All active compressed weights are counted as resident. Ordinary layer or expert offloading would need separately measured transfer and compute costs; this budget includes no offload savings.

The unchanged configuration specifies max_position_embeddings=262144 and rope_scaling=null. The upstream model card's 1M-context statement does not establish a supported context for this conversion. Neither storage-budget scenarios nor short numerical evaluations qualify long-context serving. ds4-dfm-rs implements the GB10 kernels, banks, disk KV and MTP drafting measured above and screened logits and NLL to 192K; long-running serving and long-context quality are not qualified.