Typhoon OCR V2 2B: EAGLE-3 draft head
This repository holds an optional draft head for Typhoon OCR V2 2B. It makes the model generate text faster in vLLM through speculative decoding. It does not work on its own: you always run it together with typhoon-ai/typhoon-ocr2-2b. If you do not need the speed-up, you do not need this repository.
At each step, the draft head proposes the next 3 tokens, and Typhoon OCR V2 2B checks them in a single forward pass and keeps the ones it agrees with. In our tests about 3.5 tokens come out of each step instead of 1, which makes generation 1.9 to 2.6 times faster, depending on the load.
Usage
The draft head works with vLLM only. We tested it with vLLM 0.33.0. It does not work with Transformers, llama.cpp (GGUF), Ollama or MLX.
vLLM server
vllm serve typhoon-ai/typhoon-ocr2-2b \
--served-model-name typhoon-ocr2 \
--max-model-len 40960 \
--limit-mm-per-prompt '{"image": 1}' \
--mm-processor-kwargs '{"min_pixels": 2048, "max_pixels": 16777216}' \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--speculative-config '{"method": "eagle3", "model": "typhoon-ai/typhoon-ocr2-2b-eagle3", "num_speculative_tokens": 3}'
This is the server command from the Typhoon OCR V2 2B model card with one added line, --speculative-config. Requests, prompts and outputs are the same as described there.
vLLM offline
from vllm import LLM
llm = LLM(
model="typhoon-ai/typhoon-ocr2-2b",
max_model_len=40960,
limit_mm_per_prompt={"image": 1},
mm_processor_kwargs={"min_pixels": 2048, "max_pixels": 16777216},
speculative_config={
"method": "eagle3",
"model": "typhoon-ai/typhoon-ocr2-2b-eagle3",
"num_speculative_tokens": 3,
},
)
Use the prompts, sampling settings (greedy, temperature=0) and enable_thinking=False chat template option from the main model card.
Speed
All measurements use vLLM 0.33.0 in its default compiled mode on one H100 GPU, greedy decoding and 3 draft tokens.
| Setting | Without draft head | With draft head | Speed-up |
|---|---|---|---|
| 30 pages, one request at a time | 380 tokens/s | 962 tokens/s | 2.5× |
| 30 pages, 16 requests at a time | 1,297 tokens/s | 2,403 tokens/s | 1.9× |
| MDPBench, 320 pages in batches of up to 200 | 149 s | 57 s | 2.6× |
On average 3.5 tokens come out of each step (the maximum is 4), and 82% of the proposed tokens are accepted. The 30 pages were held out from the draft head's training data. The two MDPBench runs were made on different days, so treat that speed-up as approximate. The gain is usually largest when the GPU is not fully loaded, and it becomes smaller as the number of concurrent requests grows.
Accuracy
With greedy decoding, speculative decoding is meant to keep the model's output unchanged, because every token is checked by Typhoon OCR V2 2B. In practice the output can still differ slightly. Checking several tokens at once changes the order of GPU arithmetic a little, and when two tokens are almost tied, the other one can win. Changing the batch size or switching between vLLM's compiled and eager modes has the same effect without any draft head: in our tests each of these changed the output of 5 to 8 of 30 pages.
On MDPBench (320 English and Thai pages), the scores with and without the draft head are close:
| MDPBench, vLLM 0.33.0 | Without draft head | With draft head | Change (95% interval) |
|---|---|---|---|
| Overall | 79.8 | 79.5 | −0.3 (−1.2 to +0.6) |
| English pages | 78.5 | 77.7 | −0.8 (−2.0 to +0.1) |
| Thai pages | 81.1 | 81.3 | +0.2 (−1.1 to +1.7) |
| Pages with repetition loops | 2 | 2 |
The intervals come from a paired bootstrap over pages, from one run of each. All three include zero, so we found no clear change in accuracy. The output is identical on 191 of the 320 pages, and on most of the others it differs only in a few characters. A few hard pages change more, in both directions.
Training
The draft head was trained with speculators (EAGLE-3, online training).
- Data. 20,000 page images and prompts sampled from the SFT and GRPO training data of Typhoon OCR V2, with extra weight on full-page JSON parsing and on Thai pages. The answers were generated by Typhoon OCR V2 2B itself (greedy decoding), so the draft head learns to predict this model's own outputs. After removing outputs that looped or came back empty, 19,543 conversations remained.
- Model. One transformer layer that reads hidden states from layers 2, 12 and 21 of Typhoon OCR V2 2B, with a draft vocabulary of the 32,000 most frequent tokens.
- Training. 5 epochs, learning rate 1e-4, sequences of up to 16,384 tokens, on one H100 GPU. On held-out conversations, the first, second and third proposed tokens are correct 92%, 84% and 76% of the time.
Files
| File | Content |
|---|---|
model.safetensors |
the draft head's weights (bfloat16), including a copy of the input embeddings |
config.json, config.py |
the speculators configuration that vLLM reads |
Limitations
- vLLM only, tested with vLLM 0.33.0. Other versions may need a different configuration.
- It is trained for Typhoon OCR V2 2B and does not work with Typhoon OCR V2 0.8B or any other model. For the 0.8B model, use typhoon-ai/typhoon-ocr2-0.8b-eagle3.
- Outputs can differ slightly from decoding without the draft head (see Accuracy). If you need outputs that exactly match a run without it, do not use it.
- The speed-up was measured on document pages similar to the training data. On very different inputs, fewer proposed tokens may be accepted and the speed-up will be smaller.
License
Apache 2.0, the same as Typhoon OCR V2 2B.