Ling-3.1-flash-1019 iVQ GGUF

[!WARNING] Experimental low-bit release. The routed experts, which hold most of the weights, are stored at about 1.5 bits per weight, far below common 4-bit GGUF quants. You may see failures the source model rarely shows: reasoning that loops or never reaches an answer, text that breaks mid-character (shown as �) or switches language mid-word, and weaker answers in general. This is exploratory quantization research, not a drop-in replacement for the source model.

The files can change. This line of conversions is still being researched, so any file here may be replaced by a new revision. Replacements so far: none. To keep a fixed build, download a pinned commit: hf download Baekpica/Ling-3.1-flash-1019-VQ-GGUF --revision <commit>.

Problems or odd output? Feel free to post them in the Community tab, even as rough notes. The prompt, sampling settings and runtime commit help most.

Published, verified and measured on DGX Spark. All 72 GGUF parts are uploaded; their Hub SHA-256 hashes and sizes match the locally verified release. All-12-bit expert conversion, a matched three-way quality screen and complete artifact-loader validation are finished. ds4-dfm-rs #123 serves this release on one DGX Spark (GB10), up to 1M tokens with YaRN; the updated 2K–64K measurements below use follow-up runtime e01b409d in #124 (open PR). Earlier serving and long-context records remain separately attributed.

An activation-calibrated conversion of inclusionAI/Ling-3.1-flash-1019, pinned to dddfee4bb43519a92761fbc5cefe3f0d3ff4aded. All 68 trunk routed-expert layers and the MTP routed-expert layer use 12-bit vector indices, with codebooks, assignments and row gains fitted to original-checkpoint activation statistics. Each index represents eight weights: 1.5 index bits per weight, plus gains, codebooks and container overhead.

The selected recipe is ling31-1019-ivq8-all12-q4k-embssd-v1; see recipe.json. The target is one DGX Spark with 4,096-token prefill chunks and MTP enabled. Chunk size is preserved instead of reduced to force a larger context into the memory budget.

Precision

This is the custom ling31_vq GGUF family used by ds4-dfm-rs. The 1019 recipe and host-read embedding need ds4-dfm-rs #123 or later. Standard llama.cpp/Ollama GGUF loading does not implement this representation. MTP tensors are included, calibrated and served: two drafts emit 2.39 tokens per speculative call on ds4-dfm-rs's MTP gate at e01b409d (16 cases, 1,312 tokens, 549 calls, every emitted token within the gate's 0.5 tie of a one-row decode). That tie is a regression threshold, not a measured numerical noise floor.

Measured on DGX Spark (GB10)

Measured on this release with ds4-dfm-rs e01b409d (#124, open PR). Three fresh processes per mode, one session each, 2,048-token incremental prefill and 128 forced non-EOS greedy tokens per frontier. The first 2K prefill includes first-use allocation; no separate warmup. Curves are per-frontier medians; bands include the observed min–max of all three runs, including the slower ordinary Decode process. Raw samples and exact protocol.

All 192 full-vocabulary frontier proofs are finite; repeated logits, generated tokens and acceptance cycles match within each mode. Ordinary Decode run means are 18.70 / 18.29 / 18.65 tok/s, and MTP2 means 25.81 / 25.79 / 25.73 tok/s. Busy SM clocks stay at 2,184–2,190 MHz, PSI stays zero, and minimum available memory is 5.78 GiB. The slower ordinary process remains unexplained; these data do not establish a uniform ordinary Decode gain.

The follow-up report records three Prefill and three Decode changes with separate matched controls. Ten independent short assistant cases supplement the per-round performance checks; the corpus can repeat near continuation tails, so its MTP acceptance does not establish general assistant throughput. P1 changes arithmetic: the measured chunk-size comparison covers 384 teacher-forced targets through 32K, with lower 32K top-1 agreement than the reference despite the aggregate NLL/top-1 result. Quality beyond 32K, sustained serving and 1M operation were not requalified. The earlier 1019 sweep remains archived.

The previous Ling-3.1-flash VQ v2 measured 774.1 / 17.70 (2K–64K, MTP off) and 861.0 / 23.58 (2K–32K, two drafts) under the same protocol with ds4-dfm-rs 709cf8df.

• Per decoded token the backend reads about 10.75 GB of weights (KDA projections 67%, routed experts 1.6 GB); at the 230–250 GB/s streaming ceiling that is a ~45 ms floor, against 53–54 ms measured.
• Prefill runs 4,096-row chunks: routed experts on Beam-style warp-specialized tensor-core kernels, Q4_K projections on FP16 tensor cores, KDA on a chunked tensor-core path. The optimization round records each change and its evidence.
• MTP drafts greedy requests only; thinking requests, which sample at temperature 1.0, decode without drafts.
• CUDA 13.4, sm_121a, 300–2,200 MHz clock policy.

Calibration

The original checkpoint was run over 3,732,554 tokens in 1,913 sequences, up to 32,768 tokens per sequence. Statistics cover every trunk layer and MTP, including all 512 experts of every MoE layer. MTP receives 3,730,641 shifted tokens. The corpus includes English, Korean, Chinese, Japanese, Spanish, code, raw text, thinking chats and tool conversations.

The fit uses expert-specific activation second moments, calibration-weighted Lloyd updates, weighted assignment and least-squares row-gain refitting. K4096 books are trained independently. Validation has 59,443 tokens; the separate final-test set has 52,154 tokens. Neither is used to collect calibration statistics or train the books. The three-way end-to-end screen below is complete. The recipe was fixed before this comparison; no test-set precision tuning was performed.

Size and memory

The 72 assembled GGUF parts contain 115,231,460,992 bytes (107.317661 GiB), including MTP and the BF16 embedding, versus 107.810473 GiB for the previous Ling v2 release. Packed device-resident weights total 106.118441 GiB; the 1.199219 GiB embedding file is read by row. GGUF storage is 0.492812 GiB smaller and packed device weights 1.692031 GiB smaller than the previous v2 release. Nine expert layers move from 13-bit to 12-bit; keeping the input embedding in original BF16 increases its disk size versus the old Q8_0 table, while removing its fixed device allocation.

The serving design uses a 118.3 GiB total server budget, including a 4 GiB free-memory floor. At the native 262,144-token context, FP8 latent KV, one sequence, two MTP drafts and 4,096-row prefill workspace, the allocation model predicts approximately 115.84 GiB. Counting the entire reclaimable BF16 embedding file cache as well gives approximately 117.04 GiB. These are allocation estimates, not measurements on a DGX Spark.

The source configuration has max_position_embeddings=262144 and rope_scaling=null. The upstream card advertises 1M using a YaRN extension. ds4-dfm-rs keeps the native 262,144-token context by default and serves up to 1,048,576 tokens with that recipe (-c 1048576: factor 4 over 262,144 positions, theta 6,000,000, 64 rotary dimensions, beta 32/1); the factor is fixed per session.

Historical serving measurements from #123, not rerun for this follow-up: after loading, about 12.3 GiB remain free. At the default 262K context the server keeps two persistent banks at 4,096-row chunks with 1.80 GiB spare above its 4 GiB floor. At 1M, MTP off, one bank holds 5.63 GiB of FP8 latent rows plus 276 MiB of KDA state; a 130K-token prompt prefilled at 705 tok/s with 4,096-row chunks and --mem-floor-gb 2, never below 3.0 GiB free. MTP at 1M was not measured; the allocation model above puts it over the budget.

Measured on the converted bytes

Same 27 held-out sequences, 59,416 next-token positions, on RTX PRO 6000 Blackwell. The source reads original FP8 × F32 scales; each compressed model reads its exact stored VQ/Q4_K payloads. All three use the same BF16 weight boundary, F32 routing/accumulation, FLA KDA and flash SDPA. These are reference-forward results, not ds4 serving or FP8-KV results.

Calibration reduces KL by 1.77% and improves next-token agreement by 0.37 percentage points in this screen. The gain is modest. NLL rises by 0.0428 nat versus No calibration, so the improvement is not uniform across metrics. Lower NLL than the source on this mixed fixture set is not evidence of higher general model quality; KL measures deviation from the source distribution.

No calibration disables importance weighting in both the expert VQ fit and the nonexpert Q4_K encoder. The two compressed variants have the same bit allocation and preserved tensors. Raw text, multilingual chats, thinking, tools and code are included. Per-case figures and exact scope are in quality/comparison.json.

The MTP reference predicts 59,389 shifted positions: KL 0.7261 → 0.7026 (−3.24%), next-token agreement with the original MTP 71.71% → 72.47%, NLL 1.8541 → 1.8606. These are MTP distribution comparisons, not speculative acceptance or throughput measurements.

Four cases dominate the NLL column. On tool_call_xml, swe_agentless, ds4_agent_xml and long_finance the original's reference forward scores 2.16–6.81 nat, 0.5–2.6 nat above both compressed models. Over the other 23 cases (45,268 positions) the NLLs are Original 1.3577, No calibration 1.4936 and Calibration 1.4872: both conversions cost about 0.13 nat, and calibration recovers 0.0064 nat. The KL and agreement columns include all 27 cases; per-case KL was not recorded.

The original is sensitive to evaluation batching: splitting each FFN batch in two produces KL 0.4014 and 91.29% next-token agreement against the original full-batch run. This limits strong conclusions from small changes. The reported three-way screen uses the same full-batch policy. Additional final-test and target-runtime campaigns were deferred to prioritize conversion time and cost.

Verification and use

All 214,025 original tensors are accounted for, with 107,480 stored tensors and 69 generated codebooks. Every final payload matches its converted candidate byte for byte. The Rust loader verified all 72 parts, all-12-bit books, feature selection and the host-only embedding order. Focused Ling unit tests and native C compilation passed.

Build ds4-dfm-rs #123 or later on DGX Spark (make cuda-spark) and point -m at the release directory: ./ds4-server -m <dir> serves the native 262K context, -c 1048576 1M with YaRN (MTP off), and --mtp-draft 2 enables two MTP drafts. runtime/ling31-1019.patch remains the minimal loader patch for base 73f3aa0e; see runtime/README.md and FORMAT.md.

The updated 2K–64K performance results require runtime e01b409d from #124; the earlier loader and serving support above comes from #123.

Calibration statistics, dataset provenance, tokenized splits, conversion code, receipts and quality records are retained in the private reproduction handoff. Converted candidate weights and bulky activation caches are excluded from that bucket. The official tokenizer and chat template are preserved.

Credits

inclusionAI provides the source model. The VQ representation follows the existing Ling/YoungAi family; calibration-weighted fitting follows the Beam, DeepSeek iVQ and MiniMax conversion work. Q4_K uses the GGML reference encoder. Source assets retain their original licenses; the source model card declares MIT.