Qwen3.8-Flash-Next-DR-GGUF (DR-V2-Q2_0 / Grawnch_DR_Q2_0)

Ultra-low-bit (2.45 bpw backbone) 2-shard GGUF quantization of Qwen/Qwen3.8-Flash-Next (180B MoE, qwen4exp) engineered to fit the entire active neural network inside a single 48 GB GPU (38.38 GB VRAM) while keeping the static Layer-1 N-gram lookup table (Shard 2, 28.80 GB) memory-mapped in host RAM/SSD (-lm mmap).

Unlike IQ1 / IQ2 quants that rely on non-linear lookup tables (IQ* LUTs) during matrix multiplication, every tensor in both shards uses 100% standard linear integer grids (Q2_0, Q4_0, Q6_K, Q8_0 — Zero IQ* LUTs), running natively in stock llama.cpp with 3.4x faster prompt prefill (367.0 t/s) and +34% faster single-stream decode (94.0 t/s) than IQ-grid quants.

---

1. Downstream Benchmarks & Throughput vs. Top Ultra-Low-Bit Quants

Evaluated on the full 225-problem Aider Polyglot coding benchmark (Pass@2) and the 30-problem AIME 2025 mathematics exam (Pass@1, temp = 0.6) with bounded thinking budgets (24K cap on Aider, 64K cap on AIME).

---

2. Signal Fidelity: Perplexity, KL Divergence, & Top-1 Agreement (WT2 & Ext)

Logit-level distribution fidelity measured directly against the uncompressed BF16 baseline across standard WikiText-2 (2048 context) and Extended Reasoning / Math & Code (Ext).

---

3. Detailed Metric & Architecture Breakdown (~66–68 GB Total Repo Class)

---

4. Key Architectural Highlights

• Fits in a Single 48 GB GPU (38.38 GB VRAM + ~9.6 GB KV Cache Headroom): All 48 layers of Attention/GDN, shared experts, hyper-connections, and 144 routed MoE experts live inside Shard 1 (38.38 GB, 2.45 bpw). Shard 2 (28.80 GB) contains only the static Layer-1 N-gram lookup table (per_layer_token_embd.weight), which llama.cpp streams directly from host RAM/SSD via mmap without consuming GPU VRAM.
• 5x Token Efficiency on Agentic Coding (74.7% Aider Polyglot at 40.1K Avg Tokens): Achieves 74.7% (168/225) on the full 225-problem Aider Polyglot benchmark with a 96.0% (216/225) well-formed SEARCH/REPLACE diff rate and 0 context exhaustions under a 24,576-token thinking cap — using roughly 1/5th the tokens of unconstrained 256K agentic runs.
• 2.6x Lower KL Divergence (0.2594 nats) & 83.3% on AIME 2025: Preserves 80.70% Top-1 token agreement with BF16 on Math & Code (0.9138 final-layer cosine similarity), achieving 83.3% (25/30) on AIME 2025 (temp = 0.6, 64K thinking budget) while requiring 14 GB to 30 GB less GPU VRAM than competing quants.
• Zero Lookup Tables (0 IQ* Tensors = 3.4x Faster Prefill): Because 100% of the routed experts use uniform Q2_0 linear integer grids (paired with Q8_0 shared experts & hyper-connections and Q6_K attention/GDN), prompt prefill runs at 367.0 tok/s and single-stream decode at 94.0 tok/s (~345 tok/s aggregate across 10 parallel slots at -np 10).

---

5. Files in This Repository

---

6. Quick Start (llama.cpp)

Download both shards into the same directory (making sure both share the same filename prefix, e.g., Qwen3.8-Flash-Next-DR-V2-Q2_0-00001-of-00002.gguf and Qwen3.8-Flash-Next-DR-V2-Q2_0-00002-of-00002.gguf) and pass Shard 1 to llama-server or llama-cli:

llama-server (Recommended for Coding & Math Reasoning)

llama-server \
  -m Qwen3.8-Flash-Next-DR-V2-Q2_0-00001-of-00002.gguf \
  --alias Qwen3.8-Flash-Next --host 127.0.0.1 --port 8080 \
  -ngl 999 -c 65536 -fa on -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --reasoning-format deepseek --no-reasoning-preserve

llama-cli (With Host SSD/RAM mmap for Shard 2)

llama-cli \
  -m Qwen3.8-Flash-Next-DR-V2-Q2_0-00001-of-00002.gguf \
  -lm mmap --lazy-mode on -ngl 999 -fa on \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -p "Write a fast branchless integer sorting network in C++."