Ling-3.0-tiny-UBTree — DFlash draft (GGUF, for llama.cpp)

A GGUF conversion of inclusionAI/Ling-3.0-tiny-UBTree that runs in llama.cpp as a draft-dflash speculative-decoding draft for inclusionAI/Ling-3.0-tiny.

The upstream checkpoint is a UBTree draft: a 5-layer DFlash (block-diffusion) backbone plus a learned bigram ("UBTree") selector for tree drafting. llama.cpp does not implement UBTree tree drafting/verification, so this conversion keeps the DFlash backbone and drops the 7 ubtree_selector.* tensors (the selector is not used at runtime). The result behaves as a plain DFlash block drafter: it proposes a block of tokens per step (block size 16, anchor-inclusive), which the target model then verifies.

Usage (llama.cpp)

Requires a llama.cpp build with DFlash support (--spec-type draft-dflash, upstream PR #22105 or newer). No source modifications are needed for inference — the GGUF is a standard dflash architecture file; everything is driven by metadata.

llama-server \
    -m  Ling-3.0-tiny-Q4_K_M.gguf \
    -md Ling-3.0-tiny-UBTree-dflash-bf16.gguf \
    --spec-type draft-dflash --spec-draft-n-max 6 \
    -fa on -c 32768

Notes:

--spec-draft-n-max is clamped to the draft's block size (16 → up to 15 drafted tokens). On MoE targets, smaller blocks (e.g. 6) usually win: each extra drafted token widens the expert union read during verification, so large blocks cost more than they gain.
The draft has its own KV cache, sized like the target context and typed f16 by default — on large contexts this can exceed the target's KV. Use -ctkd/-ctvd (e.g. -ctkd q4_0 -ctvd q4_0) to quantize it.
The draft shares token embeddings and the LM head with the target; they are loaded from the target model at runtime, so the draft GGUF is small (~360 MB).
Speculative decoding only pays off when the target model fits entirely in GPU memory; with CPU/mmap-resident weights, block verification has no cost advantage.
To reduce the draft footprint, it can be requantized (llama-quantize ... q8_0).

Deployment notes (measured)

AMD RX 7600 XT 16 GB, ROCm llama.cpp, target Ling-3.0-tiny-Q8_0, -fa on -c 131072 -ctk q8_0 -ctv q8_0 -np 8, greedy decoding, 3000-token completions. Draft: bf16, --spec-type draft-dflash --spec-draft-n-max 6 -ctkd q4_0 -ctvd q4_0.

Workload	Baseline	+ DFlash draft	Gain	Acceptance / mean len
Prose generation	78.8 tok/s	84.8 tok/s	+8%	0.28 / 2.7
Code generation (Python)	79.0 tok/s	127.2 tok/s	+61%	0.53 / 4.2

VRAM: 9.3 GB baseline, 13.2 GB with draft.

Conversion routine

Uses the stock llama.cpp converter (convert_hf_to_gguf.py, recent master) with a small micro-patch so it accepts the UBTree checkpoint in DFlash-only mode:

In conversion/__init__.py, add to TEXT_MODEL_MAP:
```
"Ling3UBTreeModel": "qwen",
```

In conversion/qwen.py, register the architecture on the existing DFlashModel class and filter out the selector tensors:

@ModelBase.register("DFlashDraftModel", "DFlash2DraftModel", "Ling3UBTreeModel")
class DFlashModel(Qwen3Model):
    ...
    @classmethod
    def filter_tensors(cls, item):
        name, gen = item
        if name.startswith("ubtree_selector.") or name.startswith("model.ubtree_selector."):
            return None  # UBTree bigram selector: unused in DFlash-only mode
        ...

Convert, pointing at the target model's HF directory (needed for the tokenizer and shared embeddings):

python convert_hf_to_gguf.py inclusionAI/Ling-3.0-tiny-UBTree \
    --target-model-dir inclusionAI/Ling-3.0-tiny \
    --outtype bf16 --outfile Ling-3.0-tiny-UBTree-dflash-bf16.gguf

The output contains 58 tensors (~362 MB in BF16): 5 transformer layers, the target-feature fusion projection (fc, target hidden layers 4/8/12/16/22), and norms.

Limitations

DFlash-only: the UBTree bigram selector and tree verification are not used (not implemented in llama.cpp). Full UBTree serving currently requires the (not yet publicly released) SGLang integration from inclusionAI.
The draft is tied to Ling-3.0-tiny; it cannot draft for other target models and cannot generate text on its own.
Ling-3.0-tiny is a MoE model: verifying a block of drafted tokens activates the union of their routed experts, so a verification pass reads more of the model than a single-token decode. End-to-end speedups are therefore smaller than the upstream paper's figures, which were measured with a different serving stack on datacenter GPUs.

Credits