An EAGLE-3 draft head for ai-forever/pollux-judge-32b, the 32B Russian-language LLM-as-a-judge from the POLLUX project, and for its FP8 version KaroUniform/pollux-judge-32b-FP8.
It is not a standalone model: load it as the draft model next to the target, with speculative decoding in SGLang. The head proposes tokens and the target verifies them, so outputs follow the target model up to the numerics of batched verification: about 1 % of verdicts differ from non-speculative decoding (see Quality).
| H200, SGLang, BF16 target | without the head | with the head |
|---|---|---|
| Decode, single stream | 56 tok/s | 181 tok/s (×3.2) |
| End-to-end latency p50 / p99 at 0.88 req/s | 5.3 / 13.1 s | 2.2 / 5.4 s |
| End-to-end latency p50 / p99 at 1.76 req/s | 7.9 / 22.3 s | 3.1 / 10.1 s |
| End-to-end latency p50 / p99 at 2.35 req/s | 14.7 / 45.8 s | 5.5 / 32.2 s |
| Throughput ceiling | 2.87 req/s | 3.18 req/s |
| Acceptance length (tokens per verify step) | – | 3.7–4.0 |
POLLUX-format judge prompts (2.3–2.8K input tokens), natural end of the answer, greedy, open-loop Poisson arrivals, chain drafting 5/1/6 (5 draft steps, top-k 1, 6 draft tokens), --max-running-requests 128 --chunked-prefill-size 8192. With the FP8 target at 2.32 req/s, p50 latency drops from 6.8 s to 2.6 s.
pollux-judge-32b ships repetition_penalty: 1.05 in its generation_config.json, and SGLang applies it by default. Under EAGLE verify, current SGLang applies the penalty inexactly (sgl-project/sglang#49403). As a result, about 8 % of verdicts differ from non-speculative decoding, against 0.4 % between two non-speculative runs. The fix is sgl-project/sglang#49404: with it the difference is 1.1 %, the remainder being verify-step numerics. The quality numbers below are with the fix. Until it lands, build SGLang with the PR if verdict-level reproducibility matters to you.
Speculative decoding keeps the target model's outputs up to numerical differences; the head changes only how many tokens are accepted per step. We checked verdicts end to end on 1,500 POLLUX items (greedy, repetition_penalty 1.05, FP8 target, SGLang with the fix above):
| Score changed vs. non-speculative run | MAE vs. expert scores | |
|---|---|---|
| Non-speculative, two runs | 0.40 % | 0.527 / 0.531 |
| With this head, two runs | 1.07 % / 1.07 % | 0.524 / 0.523 |
Setup: 16 concurrent requests, radix cache off; non-speculative runs also without overlap scheduling. The FP8 card measured run-to-run noise with the radix cache on (2.67 %), so the two noise levels are not comparable. Without the fix, agreement with experts did not change in our run (MAE 0.531); the fix matters for per-item reproducibility.
Verdict agreement was not measured separately on the BF16 target. Acceptance on it matches the FP8 target: 2.95–2.99 tokens per step counted over the first 3 draft tokens, on three held-out prompt sets.
LlamaForCausalLMEagle3), hidden size 5120, auxiliary target layers 1/31/60, draft vocabulary of 32,000 tokens, BF16. Embeddings are taken from the target at load time.fc (3 × 5,120 → 5,120), one decoder layer with the target's attention and MLP widths, and an LM head over the draft vocabulary, mapped to target token ids by d2t.lk_loss_type: lambda in SpecForge) in both stages.pollux-judge-32b. 5 epochs, lr 1e-4 (cosine), test-time-training length 7, maximum length 16,384 tokens.KaroUniform/pollux-judge-32b-FP8.Prompt format, scales and output format are those of the original model; see its model card.
SGLang, BF16 target:
python -m sglang.launch_server \
--model-path ai-forever/pollux-judge-32b \
--speculative-algorithm EAGLE3 \
--speculative-draft-model-path KaroUniform/pollux-judge-32b-EAGLE3 \
--speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \
--context-length 32768
FP8 target: use --model-path KaroUniform/pollux-judge-32b-FP8 and add --speculative-draft-model-quantization unquant. Without it SGLang may apply the target's quantization config to the draft head.
Sampling. Our numbers are greedy (temperature: 0). The target's generation_config.json sets temperature 0.7, top-k 70 and top-p 0.8, which SGLang applies unless the request overrides them. To sample with speculative decoding, use an SGLang build that includes sgl-project/sglang#41916 (merged 2026-11-03, after v0.5.23): it makes top-k/top-p filtering in speculative verify consistent with normal decoding.
vLLM (not tested by us). The checkpoint uses the standard EAGLE-3 layout (LlamaForCausalLMEagle3, d2t, own LM head, embeddings from the target) that vLLM's EAGLE-3 loader reads: --speculative-config '{"method": "eagle3", "model": "KaroUniform/pollux-judge-32b-EAGLE3", "num_speculative_tokens": 5}'. The speed, acceptance and repetition-penalty notes on this card are for SGLang only.
Speed was measured with SGLang nightly-dev-20261027-5884e4fe plus an earlier version of the repetition-penalty fix and sgl-project/sglang#41916; quality, with nightly-dev-20261109-842b4738 plus #49404. Both use chain drafting 5/1/6. Tree drafting (--speculative-eagle-topk > 1) was not tuned for this model.
Released under the MIT License, like ai-forever/pollux-judge-32b. LICENSE is the original MIT license of POLLUX (Copyright (c) 2025 AI Forever). The target model descends from Qwen2.5 (Apache-2.0, LICENSE-APACHE). As with the original model, users are solely responsible for reviewing and validating the model's outputs.
Training data includes material from:
deepvk/cultura_ru_edu (Apache-2.0; derived from CulturaX, whose terms follow mC4, ODC-BY 1.0, and OSCAR);t-tech/T-Wix (ODC-BY 1.0);RefalMachine/GrandMaster-PRO-MAX-Qwen3-synthetic (Apache-2.0);AtesiT/ru-llm-judge-dataset (MIT; questions from RuBQ excluded);Vikhrmodels/habr_qa_sbs (Apache-2.0);OpenAssistant/oasst2 (Apache-2.0);nvidia/HelpSteer3 (CC-BY-4.0);CohereLabs/aya_dataset (Apache-2.0).Details are in NOTICE.
Please cite POLLUX and EAGLE-3:
@misc{martynov2025eyejudgementdissectingevaluation,
title={Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX},
author={Nikita Martynov and Anastasia Mordasheva and Dmitriy Gorbetskiy and Danil Astafurov and Ulyana Isaeva and Elina Basyrova and Sergey Skachkov and Victoria Berestova and Nikolay Ivanov and Valeriia Zanina and Alena Fenogenova},
year={2025},
eprint={2505.24616},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.24616},
}
@misc{li2025eagle3,
title={EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test},
author={Yuhui Li and Fangyun Wei and Chao Zhang and Hongyang Zhang},
year={2025},
eprint={2503.01840},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2503.01840},
}