Qwen3.8-27B — NVFP4 prefiller

Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.

The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there's little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn't occupy much memory, which we achieve via offloading.

Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either.

How it works

Only the transformer stack's linear projections are disaggregated. The embedding, output head, norms and the gated-DeltaNet convolution/state parameters are shared with decode and come from the GGUF.

What it costs

The checkpoint is 12.8 GiB and never becomes resident: it is read through a two-slot ring carved out of the output head's own memory, so the ring itself adds zero device memory. Reads overlap the previous layer's compute.

Variants

One directory of blocks per decode quantization, named for the GGUF it was trained against.

Each prefiller was trained to complemet a specific decode checkpoint.

Running it

Needs a Blackwell consumer GPU — GB10 (sm_121) or RTX 50-series (sm_120). The build picks the right architecture on its own; if you pin one, it must be 120a/121a, because the block-scaled FP4 MMA is architecture-specific PTX and plain 120/121 or the 120f family target fail to assemble.

git clone https://github.com/IST-DASLab/disaggregated-llama.cpp.git llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Both halves come from the Hub, so there is nothing to download by hand. Pass the decoder and this repo; the prefill variant follows from the decoder:

./build/bin/llama-server -ngl 99 \
    -hf unsloth/Qwen3.8-27B-GGUF:IQ1_S \
    --odp-blocks ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefill

A local decoder works the same way — swap -hf ... for -m path/to/decoder.gguf.

The log tells you what it picked:

odp: taking variant 'IQ1_S' from the decoder (filename)
odp: prefill variant 'IQ1_S' from ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefill (64 parts)

Flags

ODP has non-trivial interaction with state-space models, how they're implmeneted in llama.cpp: multiple forward passes per phase to yield intermediate states, instead of pulling all the needed states from one pass. That means that each prefill can actually run full-model SSD loading multile times unless state-space cache flags are carefully tuned to ones needs.

Format

Each block is a real GGUF holding one layer's projections as GGML_TYPE_NVFP4, so gguf_dump and the viewer above show what is inside. The exact format is closer to vLLM's compressed_tensors (with static FP32 global absmax and dynamic FP8 mircoscales). NVFP4's per-tensor global scale has no slot in ggml's block layout and needs none: a scalar commutes with the matmul, so it rides in the KV as odp.scale.<name> and is applied to the result. llama.cpp's NVFP4 GEMM kernels are not particularly fast either --- the speedups above was obtained with custom FlashInfer kernels integration that was yet too ugly to release. The released for achieves speedups closer to 1.3x instead.

Files use llama.cpp's split-file convention, which is what lets one repo hold every variant:

Qwen3.8-27B-ODP-IQ1_S-00001-of-00064.gguf ... -00064-of-00064.gguf
Qwen3.8-27B-ODP-IQ1_M-00001-of-00064.gguf ...

Cite This Work

@misc{panferov2026disaggregatedquantizationspecializingllm,
      title={Disaggregated Quantization: Specializing LLM Prefill and Decode},
      author={Andrei Panferov and Maximilian Kleinegger and Sweta Priyadarshi and Tijmen Blankevoort and Dan Alistarh},
      year={2026},
      eprint={2609.26333},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2609.26333},
}