Beam-1.0-501B-A23B, EXL3 3-bit, for 2× RTX PRO 6000 Blackwell
A calibrated EXL3 quantization of Reflection's Beam-1.0-501B-A23B (501B total / 23B active parameters) and a complete recipe to serve it with vLLM on two RTX PRO 6000 Blackwell cards (96 GB each, sm_120, tensor-parallel 2 over PCIe), with Reflection's DFlash2 drafter for speculative decoding.
- 194 GB. Routed experts EXL3 3-bit, each expert calibrated on the tokens the router sends to it; attention and dense MLPs EXL3 6-bit; embeddings, LM head, norms and router as released.
- KL divergence 0.0132 as served (0.0124 offline) against the BF16 model on AtomicChat's neutral set: lower than unsloth's UD-IQ3_XXS GGUF at the same size (0.0189), and close to Reflection's official NVFP4 served on Blackwell (0.0116) at two thirds of its size.
- Agentic coding sessions: 117 tok/s decode with one session, 0.51 s median time to first token. 46K-token prompts: prefill 4,720-4,770 tok/s, decode 110-122 tok/s, 220 tok/s aggregate with four streams.
- GSM8K (first 200) 97.5 %, HumanEval 97.6 %.
Measured on two RTX PRO 6000 Blackwell Max-Q (300 W) on PCIe 5.0, no NVLink. Everything needed to rebuild the image, the
kernels and the pack itself is in recipe/.
Fidelity
Full-vocabulary KL(BF16 || quant), llama.cpp --kl-divergence scoring. Neutral = AtomicChat's eval_neutral.txt against
their published llama.cpp BF16 logits (AtomicChat/Beam-1.0-501B-A23B-GGUF-metrics),
so these rows compare directly with their GGUF table. Code / agentic = AtomicChat's code and agentic eval texts against our own
BF16 reference.
| Model | Size | Neutral KLD | Top-1 | PPL ratio | Code KLD | Agentic KLD |
|---|---|---|---|---|---|---|
| This repo, as served | 194 GB | 0.0132 | 96.65 % | 1.012 | – | – |
| This repo, offline exact | 194 GB | 0.0124 | 96.86 % | 1.011 | 0.0194 | 0.0149 |
| Reflection FP8 (official) | 503 GB | 0.0034 | 98.32 % | 1.002 | 0.0039 | 0.0122 |
| Reflection NVFP4 (official), weights only | 294 GB | 0.0065 | 97.67 % | 1.005 | 0.0084 | 0.0124 |
| Reflection NVFP4 (official), served on Blackwell (FP4 activations) | 294 GB | 0.0116 | 96.93 % | 1.010 | 0.0160 | 0.0151 |
| unsloth UD-IQ3_XXS (AtomicChat) | 197 GB | 0.0189 | 96.13 % | 1.017 | – | – |
| unsloth UD-Q2_K_XL (AtomicChat) | 187 GB | 0.0209 | 95.93 % | 1.018 | – | – |
| unsloth UD-IQ2_M (AtomicChat) | 165 GB | 0.0353 | 94.82 % | 1.034 | – | – |
"As served" is a lower bound from the server's top-128 prompt log-probs; "offline exact" scores the full vocabulary with the same weights in our reference forward pass. Details in the recipe README.
Speed
| Workload | Prefill | Time to first token | Decode, 1 stream | Decode, 4 streams |
|---|---|---|---|---|
| Agentic coding sessions (multi-turn, tool calls, prompts up to ~123K tokens) | – | 0.51 s median | 117 tok/s | 53 tok/s aggregate |
| 46K-token synthetic prompt | 4,720-4,770 tok/s | 9.6 s | 110-116 tok/s | 220 tok/s aggregate |
| 46K-token real-code prompt, fixed | ~5,000 tok/s | – | 122 tok/s | 220 tok/s aggregate |
DFlash2 with 3 draft tokens, 128K context, FP8 KV cache, a ~309K-token GPU KV pool plus a 64 GiB CPU KV tier. The recipe README has the per-length prefill table, the four-session agentic results, and every optimization step with its measured effect.
Quick start
hf download diffbot/Beam-1.0-501B-A23B-EXL3-3bpw-2x-RTX-PRO-6000 --local-dir ~/models/Beam-1.0-501B-A23B-EXL3-3bpw
hf download reflection/Beam-1.0-501B-A23B-DFlash2 --local-dir ~/models/Beam-1.0-501B-A23B-DFlash2
cd ~/models/Beam-1.0-501B-A23B-EXL3-3bpw/recipe
bash docker/build.sh && bash kernels/build.sh
API_KEY=change-me bash serve/serve.sh # OpenAI-compatible API on :8000, model name beam-1.0
Needs two 96 GB sm_120 GPUs with PCIe peer-to-peer, a CUDA 13 driver, Docker with the NVIDIA container toolkit, ~256 GB of host
RAM (pinned experts and embedding table ~100 GiB, CPU KV tier 69 GB) and ~250 GB of disk. Tool calling and reasoning use vLLM's
beam parsers.
How it fits
Beam's weights do not fit in 192 GB of VRAM with room for a KV cache, so the coldest 31 routed experts of each layer (12 %) live
in pinned host RAM and run on a side stream over PCIe, next to the VRAM experts. On top of vLLM and ExLlamaV3, the recipe adds
its own kernels: a flat decode MoE scheduler, a decode-once prefill MoE GEMM and INT8 prefill projections on INT8 tensor cores,
a flat decode GEMM for the 6-bit dense layers, and a Triton sliding-window decode attention kernel for Beam's 43 window-512
layers. See recipe/kernels/KERNELS.md.
License
The weights are a modified version of Beam-1.0-501B-A23B by Reflection, under the Apache License 2.0 (LICENSE); NOTICE lists
the modifications. The recipe's own code is MIT; the vllm-exl3 patch is AGPL-3.0-only (recipe/THIRD_PARTY_NOTICES.md).