主题

Snowflake/Arctic-LSTM-Speculator-Qwen2.5-32B-Instruct

🤗 Hugging Face 来源apache-2.0激活 32B14 GBother✓ 1 个校验和今天更新

一条命令提交

在你的模型文件夹旁边运行它。它会制作种子、将文件与 Hugging Face 比对，然后提交。你只需开始做种，并粘贴你账户中的密钥。它只读取你的文件，绝不修改。如果愿意，可以先阅读脚本。

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo Snowflake/Arctic-LSTM-Speculator-Qwen2.5-32B-Instruct ./model-folder

需要做种者 →

ArcticSpeculator

Build a fastest OSS vllm-based speculative decoding system for your own model, using ArcticTraining and ArcticInference!

We compare the throughput (tokens/s) of existing vllm-based speculative decoding systems for Llama3.1-70B-Instruct on 8xH100 as below:

method	ShareGPT	HumanEval
VLLM V1 Baseline	84.1	84.1
VLLM V1 Eagle	102.2	112.0
VLLM V1 Eagle3	77.7	85.3
VLLM V0 MLP-Speculator (IBM)	77.9	66.7
ArcticSpeculator	172.4	203.7

For more details about ArcticSpeculator and how to use it:

❄️ Using Arctic-Inference and Arctic-Training for improving real-world speculative decoding Performance (blog)
🚀 Getting started guide using ArcticTraining

We also release ArcticSpeculator checkpoints we trained with ArcticTraining to run with ArcticInference:

model	ArcticSpeculator
Llama-3.1-70B-Instruct	Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct
Llama-3.3-70B-Instruct	Arctic-LSTM-Speculator-Llama-3.3-70B-Instruct
Qwen2.5-32B-Instruct	Arctic-LSTM-Speculator-Qwen2.5-32B-Instruct
Llama-3.1-8B-Instruct	Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct