Nanbeige-4.3-33B-DFlash2 (EXL3 & FP8 Quants)

Block-diffusion speculative decoding draft model suite for Nanbeige 4.3 33B and Nanbeige 4.3 33B Heretic. Provides 1.5x to 2.2x decode acceleration.

Released Branches

Branch	Format	File Size	Notes
`4.00bpw`	EXL3 (4.0 bpw)	`531 MB`	Ultra-compact drafter for 24 GB GPUs
`6.00bpw`	EXL3 (6.0 bpw)	`754 MB`	High-fidelity ExLlamaV3 quant
`fp8`	FP8 (`e4m3fn`)	`975 MB`	Sub-1GB quantized weights
`main`	BF16	`1.96 GB`	Full precision reference export

Benchmark Speedups (RTX 5090 / RTX PRO 6000)

Evaluated with Nanbeige 4.3 33B EXL3 4.00bpw target at temperature 0.0 and 1.0 (acceptance rule: target sample then match):

Domain	Mean Accepted Tokens / Step	Decode Speed	Speedup vs Plain
Agentic Coding	2.64 tokens	143.0 tok/s	2.16x
Code Generation	2.21 tokens	119.6 tok/s	1.79x
Math and Reasoning	2.21 tokens	119.9 tok/s	1.78x
Conversational Chat	1.52 tokens	82.3 tok/s	1.22x
Multilingual (Chinese)	1.45 tokens	78.6 tok/s	1.18x
Overall Average	1.86 tokens	101.1 tok/s	1.51x

VRAM Planning

Speculative decoding pairs the target base model with the drafter in VRAM:

Target Model Quant	Drafter Quant	Target VRAM	Drafter VRAM	Total Model VRAM	Free VRAM on 32 GB GPU	Free VRAM on 24 GB GPU
EXL3 4.00bpw	4.00bpw	18.6 GB	0.53 GB	19.1 GB	12.9 GB	4.9 GB
EXL3 4.00bpw	FP8	18.6 GB	0.98 GB	19.6 GB	12.4 GB	4.4 GB
EXL3 4.00bpw	BF16	18.6 GB	1.96 GB	20.6 GB	11.4 GB	3.4 GB
EXL3 3.50bpw	4.00bpw	16.5 GB	0.53 GB	17.0 GB	15.0 GB	7.0 GB

TabbyAPI Quickstart

Add the drafter configuration to your tabby_config.yml:

model:
  max_batch_size: 1
  cache_mode: '8,8'
  max_seq_len: 65536

draft_model:
  draft_mode: model
  draft_model_name: Nanbeige-4.3-33B-DFlash2

ExLlamaV3 Python Quickstart

import os
os.environ['EXL3_GRAPHS'] = '1'
import torch
from exllamav3 import Config, Model, Cache, CacheLayer_quant, Tokenizer, Generator

target_cfg = Config.from_directory('./Nanbeige-4.3-33B-Heretic-EXL3-4.0bpw')
target_model = Model.from_config(target_cfg)
target_cache = Cache(target_model, max_num_tokens=65536, layer_type=CacheLayer_quant, k_bits=8, v_bits=8)
target_model.load(device='cuda:0', max_batch_size=1)
tokenizer = Tokenizer.from_config(target_cfg)

draft_cfg = Config.from_directory('./Nanbeige-4.3-33B-DFlash2')
draft_model = Model.from_config(draft_cfg)
draft_cache = Cache(draft_model, max_num_tokens=65536)
draft_model.load(device='cuda:0', max_batch_size=1)
draft_model.attach_to(target_model)

generator = Generator(
    model=target_model,
    cache=target_cache,
    tokenizer=tokenizer,
    draft_model=draft_model,
    draft_cache=draft_cache,
    num_draft_tokens=7,
    dynamic_draft_tokens=False,
    max_batch_size=1,
    record_draft_stats=True,
)

Architecture Details

Architecture: DFlash2DraftModel
Layers: 3
Hidden dimension: 7168
Intermediate dimension: 8192
Attention heads: 28 (KV heads: 4)
Block size: 8 (7 speculative proposals + 1 anchor)
Target tap layers: [1, 25, 49] (shift 0)
Vocabulary size: 166,912

Attribution & Provenance

Draft Model Base Checkpoint: ProCreations/Nanbeige4.3-33B-DFlash2 (trained by ProCreations under Apache-2.0).
Target Base Model: Nanbeige/Nanbeige4.3-33B by Nanbeige Team.
Uncensored Target Model: Luni/Nanbeige-4.3-33B-Heretic.
DFlash Architecture: Z-Lab DFlash (Jian Chen, Yesheng Liang, Zhijian Liu).
Compatibility: Draft proposals are evaluated against the target model's logits at runtime, ensuring lossless verification on both Stock Base and Heretic without refusal leakage.

Luni/Nanbeige-4.3-33B-DFlash2