Spark-X2.6-4B with a Causal Encoder-Decoder (CED) split: experimental retrofit

Update 2026-10-29: v2, split moved from layer 18 to layer 24 (recommended)

v1 lost a lot of long-context retrieval. The reason turned out to be the split point: the decoder's global layers (23, 27) need information that the layer-18 state simply doesn't contain yet (a closed-form fit from H_18 to the original layer-23/27 values leaves 37-48% error, and no projector training fixed it; from H_24 it's 6%). v2 moves the split to layer 24: layers 0-23 = encoder (all original weights, including global layer 23), layers 24-35 = decoder, whose 3 global layers (27, 31, 35) read K/V projected from RMSNorm(H_24).

v2 is not trained at all. The 6 projection matrices are a closed-form ridge-regression fit of the original model's K/V from H_24 (about 0.5M tokens of PG19 / code / chat / synthetic retrieval); there is no LoRA and every other weight is the original.

The trade: more of the model runs over the whole prompt (24 of 36 layers instead of 18), so prefill saves about a third instead of about half; in exchange retrieval comes most of the way back to the original. Decode speed is the same as the original either way. Long-context items are thinking-off, greedy, up to 32K tokens; RULER variable tracking is included with the answer prefix pre-filled as in the official setup. llama.cpp prefill on an Intel Arc Pro B70 (Vulkan, Q4_K_M, llama-server two-phase prefill with replay 1024, ub 512, warm; prompt of repeated token ids, timing only), tokens/s at 4K / 8K / 16K / 32K: original 2867 / 2503 / 1942 / 1332, v1 4234 / 4147 / 3535 / 2595 (1.48 / 1.66 / 1.82 / 1.95x), v2 3616 / 3358 / 2710 / 1907 (1.26 / 1.34 / 1.40 / 1.43x). The PyTorch speedups in the table use the same reference implementation for all variants (the GGUF parity is: mean KL 3.5e-4 to 2.3e-3 vs the PyTorch path, within about 2x of the plain-GGUF-vs-PyTorch noise floor on the same sequences).

v2 files: spark-ced24-Q8_0.gguf, spark-ced24-Q4_K_M.gguf (GGUF key spark2_5.ced.split = 24, same llama.cpp patch as v1, same bartowski imatrix), ced_weights_split24.safetensors (the 6 projections, bf16, 31 MB; demo.py reads the split from the file). The v1 files below are unchanged.

---

v1 (2026-10-26): split at layer 18, trained

Experimental research artifact, not a drop-in model. Runs in PyTorch (demo.py) and in a llama.cpp fork (GGUFs below); stock llama.cpp and vLLM cannot load it.

This takes the finished XHToken/Spark-X2.6-4B (36 layers, sliding-window layers with a global-attention layer every 4th layer) and converts it to the CED layout of DeepSeek-V4.1-Flash (arXiv 2609.19969, sec. 2.2 and 3.2.2) without pre-training anything from scratch:

• layers 0-17 = causal encoder (layers 0-13 are the original weights, 14-17 carry a small LoRA), layers 18-35 = decoder;
• the global-attention K/V of the 5 decoder global layers (19, 23, 27, 31, 35) are projected from the encoder's last hidden state with layer-dependent weights instead of from each layer's own hidden state: C_l = RMSNorm(H_18) W_l (the paper writes C_l = H_{L/2} W_l; the gain-free RMSNorm on the input is my addition so the projections can be initialised from the layers' own weights);
• decoder sliding-window K/V stay layer-wise and are rebuilt by bounded replay of the last prompt tokens.

Prefill runs the encoder (18 layers) over the whole prompt and the decoder (18 layers) over only the last replay tokens: close to half the prefill compute. Decode compute and the KV-cache size are unchanged (exact CED keeps one cache per layer; KV compression is CSA2's job, not CED's).

What was trained

Only a small number of weights, by distillation against the original model (KL on its own logits + normalised per-layer hidden-state MSE; no labels, no teacher other than Spark itself), about 50M training tokens in total on one local B70 (~6 h):

1. closed-form ridge init of the 10 new K/V projections (5 layers x K,V, 26M parameters) from H_18 onto the original layers' K/V;
2. training of those projections;
3. • LoRA (rank 16) on all 18 decoder layers;
4. • LoRA (rank 16) on encoder layers 14-17 (data: PG19 books, GitHub code, WildChat);
5. • 300 more steps with the original model's own sampled answers to GSM8K-train questions and general prompts mixed in (each conversation as its own sequence, full replay).

Encoder layers 0-13 are bit-identical to the original. The LoRA can be folded into the weights (Spark.merge_lora), after which decode runs at the original's speed.

Quality: distillation fidelity

85 held-out 8192-token sequences (44 PG19, 24 code, 17 chat) never used for training or for choosing a checkpoint. KL(original || CED) on the last 512 positions, lower is better:

• Replay of 1024 tokens (2x the 512 window) is lossless (KL 0.000 on top of the K/V approximation); the paper-style replay of exactly one window (512) costs 0.05-0.08 KL by itself. Use replay=1024.
• Cross-entropy on the last 1024 tokens, original -> CED (PG19 / code / chat): 2.676 -> 2.705 / 0.961 -> 0.993 / 2.089 -> 2.017. Top-1 agreement with the original: 0.885 / 0.922 / 0.889.

Next to ordinary quantization

KL to the unquantized model for the usual GGUF quants (bartowski imatrix quants vs his bf16 GGUF, llama.cpp --kl-divergence on CPU, 4K-token chunks, same text domains) next to CED. Not the same positions as the CED numbers (CED: last 512 of 8K sequences), so read it as a rough placement:

CED costs about as much as an IQ4_XS-to-IQ3_M quant. It saves prefill compute, not memory, and the two should stack (errors add up).

Task check: GSM8K, 100 questions

Greedy, chat template, no thinking, the same PyTorch code path for both: original 89/100, CED 87/100 (6 questions only the original solved, 4 only CED). A 2-question gap on 100 is within noise.

Speed (same reference implementation for both, bf16, LoRA merged, steady-state median, Intel Arc Pro B70)

Prefill:

The reference implementation is plain PyTorch with chunked attention, so absolute tokens/s are not comparable with optimised engines; the ratio is what carries over. The asymptote is ~2x. Decode at 8K context: original 52.4 ms/token, CED 52.9 ms/token (LoRA merged). CED is a prefill optimisation, not a decode one.

Retrieval ("needle in a haystack" with 4 near-duplicate distractors, 20 trials per length, greedy)

llama.cpp (GGUF)

spark-ced-Q8_0.gguf (4.4 GB) and spark-ced-Q4_K_M.gguf (2.6 GB) have the LoRA merged in and carry the 10 K/V projections as extra tensors (blk.N.ced_k, ced_v; GGUF key spark2_5.ced.split). They need llama.cpp with the CED patch (llama.cpp-ced.patch in this repo, against upstream commit a0e6dc738; git apply it and build as usual, e.g. -DGGML_CUDA=ON or -DGGML_VULKAN=ON); stock llama.cpp refuses to load them (tensor-count error). The patch adds the CED graph, the converter and a llama_set_ced API and does nothing for models without CED tensors. llama-server -np 1 does the two-phase prefill by itself (replay 1024, LLAMA_CED_REPLAY=0 turns it off); other slot counts fall back to normal prefill. The quants use bartowski's imatrix, not one of my own.

Logits match the PyTorch CED within the bf16 noise floor (mean KL 3e-4 to 8e-4, CPU and Vulkan); with the decoder sliding-window mask deliberately removed the same test shows KL 0.34, so it does catch a broken graph. KL to the original bf16 GGUF (mean over PG19 / code / chat, llama.cpp --kl-divergence): CED bf16 0.067, CED Q8_0 0.068, CED Q4_K_M 0.114 (plain Q4_K_M: 0.044).

Prefill, tokens/s, Q4_K_M, warm, ub 512, -ngl 99, stock = XHToken's official Q4_K_M:

Decode (4090, tok/s at 4K / 32K / 128K context): stock 219 / 175 / 93, CED 226 / 177 / 94. The 4090 numbers are from a rented host with an i9-14900K; an earlier run on a slow server CPU was CPU-bound (GPU 48-64% busy) and showed almost no gain, so prefill speedups on a weak host will be smaller. The B70 numbers were measured on a build that also has my own Intel Xe2 Vulkan kernel changes (not in the patch here); the stock column is the same build without CED, so the ratio is CED's.

Usage

python demo.py /path/to/Spark-X2.6-4B ced_weights.safetensors --replay 1024 --compare

spark_ced.py is a self-contained PyTorch re-implementation of Spark (checked against the reference code and an fp32 CPU run) with the CED path. ced_weights.safetensors holds the 10 K/V projections and the LoRA deltas (bf16, 89 MB); the base weights come from the original repo. In code: proj, lora = load_ced_weights(...), S.merge_lora(lora), then S.ced_prefill(ids, proj, caches, tail=1024) and S.ced_step(...) with lora=None.

Caveats

• Not an official or endorsed variant of Spark or DeepSeek; the architecture is from the DeepSeek-V4.1-Flash paper.
• The paper trains CED from scratch with a jointly trained encoder; this retrofit adapts a finished model with LoRA and distillation, so it is lossy (see tables). Greedy outputs diverge from the original after a few dozen tokens; a multi-step arithmetic prompt can come out wrong where the original is right.
• Evaluation is distillation fidelity (KL/CE), 100 GSM8K questions and a synthetic retrieval test, not task benchmarks. The PyTorch speed table above comes from a reference implementation, not an inference engine; the llama.cpp numbers are above.
• GGUF speed numbers use a prompt of repeated token ids (timing only; quality is measured separately); the server path has been checked for identical greedy output on CPU only.
• CED GGUF Q4_K_M is noticeably lossier than plain Q4_K_M (KL 0.114 vs 0.044); Q8_0 adds nothing on top of CED.

Credits

Made by @imdariotoo. I post my local-model releases on X, so follow there for the next ones.