Spark-X2.6-4B EAGLE-3 draft head

A one-layer EAGLE-3 draft head (218M parameters) for XHToken/Spark-X2.6-4B. It proposes tokens that the 4B model verifies, so speculative decoding runs faster while the output stays exactly what the target model would have produced.

There was no public draft head for Spark, so this is a from-scratch one. It is provided so you can combine it with whatever target GGUF you already use.

This is v8, the current release: trained on a mixed code + prose + conversation corpus with verifier-argmax distillation and the EAGLE-3 training-time test.

⚠️ Requires a one-line llama.cpp patch

llama.cpp's EAGLE-3 draft consumes the input to three target decoder layers, exposed per architecture as res->t_layer_inp[il]. Spark's arch (spark2_5) does not set it, and the runtime aborts with:

GGML_ASSERT(t_layer_inp[il] != nullptr)

Apply llama.cpp-spark2_5-layer-inputs.patch (included here) to src/models/spark2-5.cpp and rebuild:

     for (int il = 0; il < n_layer; ++il) {
+        res->t_layer_inp[il] = inpL;
+
         ggml_tensor * inpSA = inpL;

Tested against llama.cpp 0.7.0-dev (commit f5027fb).

Quick start

llama-server \
    --model Spark-X2.6-4B-Q8_0.gguf \
    --draft spark-eagle3-v8.gguf --spec-type draft-eagle3 \
    --spec-draft-n-max 3 \
    -ngl 99 --ngld 99

Where the 4B target does not fit in VRAM, keep it on the CPU and put only the head on the GPU — the head costs ~0.5 GiB including a 4k context:

llama-server \
    --model Spark-X2.6-4B-Q8_0.gguf \
    --draft spark-eagle3-v8.gguf --spec-type draft-eagle3 \
    --spec-draft-n-max 3 \
    --ngl 0 --ngld 99 --threads 4 -c 4096 -np 1

--spec-draft-n-max: 3 for prose, 2 never, 1 for code. Prose depth-1 acceptance is 0.75, so a pass returning ~3.2 tokens beats verifying four; code depth-1 is ~0.15, so only n=1 helps there.

Avoid 2 when the target is on the CPU. It is not a middle setting: in llama.cpp 0.7.0-dev a pass at n_max 2 costs ~350 ms against ~182 ms at 1 and ~217 ms at 3, so it lands below both (prose 1.28x vs 1.84x/2.57x). The same value behaves normally with the target in VRAM, so this is a CPU-side code-path artifact.

Specs

Note for anyone loading the safetensors by hand: d2t is stored as deltas (convert_hf_to_gguf.py adds arange to recover absolute ids). Exporting absolute ids fails with EAGLE-3 d2t target ids out of range for target vocab size 131072.

Training

Three choices carried the gain, each measured against the previous version:

Label on the verifier, not the corpus. Acceptance is draft argmax == target argmax, so wherever the corpus token is not what Spark would emit, ordinary cross-entropy teaches a position the draft can never win. Relabelling lifted prose acceptance 0.49 → 0.81 at identical cost.

Training-time test. The head is also fed its own prenorm output back as the next step's g, with a second loss there, so it learns to recover from its own drafts. This raised prose depth-1 acceptance 0.50 → 0.70 and is what makes n_max > 1 pay.

Conversational data. Instruct prose rendered through Spark's own template lifted chat-format acceptance 0.299 → 0.325 (0.377 with thinking disabled). Templated code in the same quantity moved it 0.001, so the win is the register, not the markers.

Results

Depth-0 acceptance, greedy, 96-token generations, target in VRAM:

Throughput, single RTX 3090 + i7-6700K (n=3 unless noted):

At n=1 the CPU placement gives 1.88x on prose and 1.48x on mixed prompts. The gain scales with how slow the verifier is: with the target in VRAM a token costs ~7 ms, so the draft's fixed per-pass cost stops being negligible.

Limitations

• Chat format is still weaker than raw text. 0.325 against 0.464 on the same prompts: the template opens a thinking channel, and the corpus contains no reasoning traces, so thinking-mode acceptance lags thinking-off (0.377).
• Narrow out of domain. 0.92 in-domain but 0.46 on mixed prompts. A public EAGLE-3 head for Qwen3-4B scored 0.53–0.64 on the same mixed/code prompt sets.
• Code is the weakest domain, and p1 ≈ 0.15 means only n_max 1 is useful. Code p0 is 0.415 here against 0.437 for the previous release; that difference is within the ±0.03 noise of a 3-prompt evaluation and coincided with cutting the code share from 12M to 10M tokens, so treat it as unresolved rather than real.
• Two-layer capacity is blocked upstream: convert_hf_to_gguf.py only detects EAGLE-3 at num_hidden_layers == 1, and llama.cpp's eagle3 graph builds one layer, so a deeper head needs patches on both sides.
• Small evaluation. 12 hand-written + 12 WikiText + 3 code prompts, in two prompt forms; the mixed and code figures carry roughly ±0.03 uncertainty.
• Spark only. The head reads Spark's hidden states at layers 2/18/33 with Spark's tokenizer; it will not work with another model.

Files

To rebuild the GGUF from the safetensors (needs a local copy of the Spark config and tokenizer, which the converter reads but does not instantiate):

python3 convert_hf_to_gguf.py v8 --target-model-dir /path/to/Spark-X2.6-4B \
    --outfile spark-eagle3-v8.gguf --outtype bf16

License

Apache-2.0, following the base model. Training data: codeparrot-clean (permissively licensed GitHub Python), Magicoder OSS-Instruct (MIT), UltraChat-200k (MIT), WikiText-103 (CC BY-SA 3.0) — the last is share-alike, which is worth knowing if you redistribute this commercially.

Citation

If this is useful, cite the EAGLE-3 paper for the architecture (Li et al., 2025) and link back here for the weights.