Ripple-65B Preview — EXL3 4.0 bpw
EXL3 quant of ricursiveAI/Ripple-65B-Preview, produced locally with exllamav3 1.6.6 plus the feat/ricursive branch on 5× RTX 3090.
Ripple is a 65B dense model: 80 layers, 5120 hidden, 40960 FFN, 80 Q / 8 KV heads, head_dim 128, 100352-token OLMo vocabulary, 2M position limit. It uses a 4:1 sliding-window (1024) / global NoPE attention pattern, Gemma-3N-style PeriNorm blocks, sigmoid-gated attention, per-head Q/K RMSNorm, SwiGLU clamp ±20, embedding scale √5120, and a learned per-layer value residual (value_res_lambda, a sigmoid-gated blend of the scaled layer-0 embedding into V).
| Quant method | EXL3 (exl3, trellis, mul1 codebook, out_scales: auto) |
| Quant version | 1.6.6 |
| Body bitrate | 4.0 bpw |
| Head bitrate | 6 bpw (lm_head) |
| Embedding bitrate | 16 bpw |
| Calibration | 250 rows × 2048 cols |
| Size on disk | 33.3 GB, 4 shards |
| Architecture | RicursiveForCausalLM |
| Base model licence | Apache 2.0 |
Requirements
This checkpoint needs the value-residual path, which is not in upstream exllamav3 1.6.6. Use the feat/ricursive branch of auryn-macmillan/exllamav3 (or a build that includes it). It works directly with exllamav3 or through TabbyAPI. It is not usable with transformers, vLLM, or llama.cpp.
The branch adds RicursiveForCausalLM, carries the layer-0 embedding seed through calibration, feeds only V from lerp(seed, normed_hidden, sigmoid(lambda)), and declines BC attention for value-residual modules.
Tested configuration (4× RTX 3090, TabbyAPI)
ripple-production-4card.yml in the serving tree is the profile that was actually used:
model:
max_seq_len: 262144
cache_size: 2097152 # 8 × 256K slots
cache_mode: Q4
max_batch_size: 8
chunk_size: 4096
gpu_split: [19.0, 19.0, 19.0, 19.0]
gpu_split_auto: false
Cards 0,1,2,4; GPU 3 is left to parakeet-stt and the Gemma-4 compactor. Load is 24 s warm / 34 s cold, about 18.5 GB per card, with 4.3–6.6 GB free. The draft model is disabled.
Measured performance (4× 3090, Threadripper PRO 3975WX, 247 GB RAM)
| shape | per stream | aggregate decode | TTFT p50 / max |
|---|---|---|---|
| 2K × 1 | 20.5 T/s | 20.5 | 3.4 s |
| 2K × 8 | 11.8 T/s | 94.6 | 21.3 s |
| 12K × 1 | 19.9 T/s | 19.9 | 17.1 s |
| 12K × 8 | 8.3 T/s | 66.4 | 121.1 / 148.8 s |
| 49K × 1 | 19.6 T/s | 19.6 | 55.7 s |
| 49K × 2 | 15.9 T/s | 31.7 | 150.9 s |
| 128K needle | 22.5 T/s | 22.5 | 235.5 s (chunk 2048) |
| 245K needle | 18.9 T/s | 18.9 | 495.6 s (chunk 4096) |
The 245K request returned the right code from a 240,942-token prompt. A 12K multi-turn session cached the prefix: TTFT 17.2 s cold, then 1.46 s and 1.51 s on the next two turns, at ~20.2 T/s.
Prefill is the serialized cost. Chunk 4096 cut 49K TTFT from 76.0 s to 55.7 s (27%) and 12K from 18.1 s to 17.1 s; chunk 8192 does not load.
Quality
4.0 bpw is coherent: the code prompt produced a correct iterative Fibonacci function with tests, 17*23 returned 391, and the 245K needle came back with the exact code. No obvious 4 bpw damage in these checks.
DFlash2
The native 4B DFlash2 drafter is converted separately at 4.0 bpw. At this target bitrate its acceptance is only 3–6%: 12K × 1 is +6% (21.6 T/s), 12K × 8 is worse (7.2 T/s/stream). It is not attached to this profile. Keep it for a future higher-bitrate target.
Recipe
The conversion command and bitrate summary are included as recipe/ripple-4.0bpw.yaml. The upstream model card is kept as UPSTREAM_README.md.