DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference
Overview
DART is a new speculative decoding approach for Large Language Models (LLMs) inference, which is inspired by diffusion-based Large Language Models (dLLMs). DART surpasses EAGLE3 by 30% on average, achieving up to 65% improvement on certain code-centric workloads.
DART's drafting requires only a single forward pass of a single transformer layer and a fast cpp-based tree search to build the draft token tree, resulting in extremely low drafting cost while preserving relatively high $\tau$ (Average Acceptance Length).
Key Features
- Fast drafting Forward: Produces multiple logits simultaneously with 1 forward of 1 layer.
- Fast Tree Search: Uses n-gram–based tree search to build the final draft tree (C++ based).
- Low Drafting Cost: Results in extremely low drafting cost for efficient inference.
- relatively high $\tau$: Average Acceptance Length is competitive with EAGLE3.
Quick Start
Installation
git clone https://github.com/fvliang/DART.git
cd DART
curl -LsSf https://astral.sh/uv/install.sh | sh (optimal, if you don't have a uv)
uv sync
uv pip install -e .