Ornith-1.5-35B-A3B-DFlash-GGUF

GGUF conversion of ornith-ai/Ornith-1.5-35B-A3B-DFlash for llama.cpp (--spec-type draft-dflash). The weights are unchanged apart from the format and quantization. For details about the model, see the original repository.

Conversion

Source: model.safetensors from ornith-ai/Ornith-1.5-35B-A3B-DFlash (SHA-256 b9d55100f43e1a2bd56e287ac11309086ccb827ae5d6f546dc9197c2c75b34f6). Converted with llama.cpp commit 3057bb6:

python convert_hf_to_gguf.py Ornith-1.5-35B-A3B-DFlash \
  --target-model-dir Ornith-1.5-35B-A3B \
  --outtype bf16 --outfile Ornith-1.5-35B-A3B-DFlash-BF16.gguf

Usage

Load it as the draft model together with an Ornith-1.5-35B-A3B GGUF:

llama-server \
  -m Ornith-1.5-35B-A3B-Q4_K_M.gguf \
  -md Ornith-1.5-35B-A3B-DFlash-Q8_0.gguf \
  --spec-type draft-dflash

It can also be used with models based on Ornith-1.5-35B-A3B, such as Tiel-Coder-35B-A3B.

Measurements from one iGPU are in benchmark-report.md.

Known issues

• On an AMD iGPU with Vulkan, older llama.cpp builds (e.g. 4fea119) could hang the GPU with draft-dflash. Use a recent build. See also llama.cpp issue #27458.

License

MIT, same as the original model. All credit for the model goes to Ornith AI.