Ripple-65B Preview DFlash2 — EXL3 4.0 bpw

EXL3 quant of ricursiveAI/Ripple-65B-Preview-DFlash2, produced locally with exllamav3 1.6.6 plus the feat/ricursive branch on 5× RTX 3090.

DFlash2 is Ripple's native speculative drafter: 4.04B parameters, 5 layers, hidden 5120, 80 Q / 8 KV heads, head_dim 128, SwiGLU FFN 40960, 100352 vocabulary. It drafts blocks of 16 tokens, retains 32 candidates per position, uses a rank-256 selector, and taps target hidden states at layers 1, 20, 39, 58, 77. Attention is four sliding layers (4096 window) plus one full-attention NoPE layer. Configured maximum positions: 131072.

Requirements

This is a draft model, not a standalone chat model. It requires the Ripple target and its tokenizer, chat template, token embeddings, and output head. Use the feat/ricursive exllamav3 branch for the target; the draft needs DFlash2 support in the serving runtime.

TabbyAPI draft config used in the test:

draft_model:
  draft_mode: model
  draft_model_dir: /home/auryn/models/exl3
  draft_model_name: Ripple-65B-Preview-DFlash2-4.0bpw
  draft_gpu_split: [0, 0, 0, 4.5]
  draft_num_tokens: 15
  draft_max_rows: 16
  draft_cache_mode: "4,4"

Measured behavior

Against the 4.0 bpw Ripple target, acceptance is only 3–6%. At 12K × 1 it is +6% (21.6 T/s); at 12K × 8 it is worse (7.2 T/s/stream). It is not recommended attached at this target bitrate. The 8 × 256K Q4 production pool also does not fit the draft: 2.1 GB of weights plus its Q4 cache exceed the free memory on GPU 0 and GPU 4.

Keep this checkpoint for a future higher-bitrate target or a cleaner drafter/target pairing.

Recipe

The conversion command and bitrate summary are included as recipe/ripple-dflash2-4.0bpw.yaml. The upstream model card is kept as UPSTREAM_README.md.