Ornith-1.5-35B-A3B-Abliterated — ROCmFP4 COHERENT (ftype 102)
The model here is not our work. The abliteration is huihui-ai's
(Huihui-Ornith-1.5-35B-A3B-abliterated); the BF16 GGUF we quantised from is
PocketAiHub's; the base model is
ornith-ai's. This repository adds only the ROCmFP4/ROCmFPX
quantisation ladder, the dual-backend build, and the measurements. Go star their repos.
**The Vulkan tier. 64.51 tok/s decode on a Radeon 8060S — the fastest of six quantisations we measured
of this model, and +14.4 % over the same file on ROCm.**
Single-file release: Q4_0_ROCMFP4_COHERENT, 18.5 GiB, 4.58 BPW, protected q6_K head and q6_K token
embeddings. One binary, two backends — the backend is a runtime -dev flag, not a rebuild.
Why this tier
| | |
|---|---|
| Decode (Vulkan) | 64.51 tok/s — highest of all six tiers built from this source |
| Decode (ROCm) | 56.38 tok/s |
| Vulkan uplift | +14.4 % — the largest backend gain of the family |
| Prefill (ROCm) | 395 tok/s |
| Size / BPW | 18.5 GiB / 4.58 |
| output.weight | q6_K (protected, verified by read-back) |
| token_embd.weight | q6_K — the only 4-bit tier here with q6_K embeddings |
| VRAM/GTT at 32K ctx | ~20–21 GiB |
This is the tier to pick if you run chat or agentic loops on Vulkan. It is the only 4-bit variant in
this family that keeps both the output head and the token embeddings at q6_K, and it posts the highest
decode number we measured on this hardware.
Quick start
# Vulkan — what this tier is for (64.51 tok/s)
llama-server -m Ornith-1.5-35B-A3B-Abliterated-Q4_0_ROCMFP4_COHERENT.gguf \
-dev Vulkan0 -fa on -ngl 999 -fit off -np 1 \
-ctk q8_0 -ctv q8_0 -c 32768 -b 8192 -ub 2048 -t 4 --poll 100 --jinja \
--repeat-penalty 1.05 --repeat-last-n 256 \
--host 0.0.0.0 --port 8080
# HIP / ROCm — SAME FILE, only -dev changes. Better prefill (395 vs 249 tok/s).
llama-server -m Ornith-1.5-35B-A3B-Abliterated-Q4_0_ROCMFP4_COHERENT.gguf \
-dev ROCm0 -fa on -ngl 999 -fit off -np 1 \
-ctk q8_0 -ctv q8_0 -c 32768 -b 8192 -ub 2048 -t 4 --poll 100 --jinja \
--repeat-penalty 1.05 --repeat-last-n 256 \
--host 0.0.0.0 --port 8080--repeat-penalty 1.05 --repeat-last-n 256 is recommended, not decorative: with every repetition control
at its default (repeat_penalty=1.0, dry_multiplier=0.0) a greedy decode can enter an unbounded
repetition loop on agentic tasks. We measured this on a sibling model and it cost ~7–11 % throughput to
prevent. ⛔ Do not reach for DRY sampling on checksum/hash work — we measured DRY corrupting a sha256
the model was asked to reproduce.
✅ **This exact tier was truncation-tested on both backends and passed: 0 in 48 on Vulkan, 0 in 48 on
ROCm**, every generation reaching the full 400 tokens. The sibling 9B fails this same test on
Vulkan (8/48, 17 %), which is why it was worth running. See Vulkan stability on gfx1151 below.
⛔ You need a ROCmFPX build with Vulkan enabled — stock llama.cpp will NOT load this file
ROCmFP4 uses ggml tensor types 100–119; upstream llama.cpp's table stops at 43:
gguf_init_from_reader: tensor '...' has invalid ggml type 101. should be in [0, 43)git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
cmake -B build-hipvk -G "Unix Makefiles" \
-DCMAKE_MAKE_PROGRAM=/usr/bin/gmake \
-DCMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang \
-DGGML_HIP=ON -DGGML_VULKAN=ON -DGGML_NATIVE=ON \
-DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc \
-DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build-hipvk -j16 --target llama-server
build-hipvk/bin/llama-server --list-devices # must list BOTH ROCm0 and Vulkan0Traps: no Ninja on many Strix boxes → Unix Makefiles + gmake · glslc is not auto-detected → pass
-DVulkan_GLSLC_EXECUTABLE · a stale CMakeCache poisons a generator switch → rm -rf the build dir ·
-j32 OOM-killed a process on a 128 GB box → use -j16.
Measured — this tier, both backends
Ryzen AI Max+ 395 (Strix Halo) · Radeon 8060S gfx1151 · 128 GB unified · ROCm 7.2.4 · Ubuntu 24.04.
-c 32768, -ctk/-ctv q8_0, 4 fixed prompts, temperature 0, top_k 1, 400 tokens,
cache_prompt:false, warm-up discarded, median of 3, box idle.
| Backend | code | prose | reason | json | prefill | GTT |
|---|---:|---:|---:|---:|---:|---:|
| Vulkan0 | 64.51 | 62.63 | 64.27 | 64.03 | 249 | 20 GiB |
| ROCm0 | 56.38 | 56.28 | 56.15 | 56.32 | 395 | 21 GiB |
| Δ | +14.4 % | +11.3 % | +14.4 % | +13.7 % | −37 % | |
The crossover is a trade, not a free win. Vulkan wins decode on every workload; ROCm wins prefill by
59 %. Decode-bound work (chat, agents, warm cache) → Vulkan. Prompt-bound work (long context, RAG, cold
start) → ROCm. Same file either way.
How this tier compares to the rest of the family
Full six-tier table, both backends: Ornith-1.5-35B-A3B-Abliterated-ROCmFP4-GGUF
| Tier | ftype | Size | Vulkan decode | ROCm decode | ROCm prefill |
|---|---|---:|---:|---:|---:|
| COHERENT (this repo) | 102 | 18.5 GiB | 64.51 | 56.38 | 395 |
| STRIX_LEAN | 106 | 17.5 GiB | 62.69 | 56.59 | 423 |
| FAST | 103 | 17.4 GiB | 62.30 | 57.35 | 426 |
| Q8_ROCMFPX | 111 | 33.4 GiB | 46.82 | 42.28 | 331 |
| Q8_ROCMFPX_AGENT | 115 | 33.9 GiB | 46.17 | 41.67 | 337 |
| Q6_ROCMFPX_AGENT | 114 | 30.1 GiB | 43.19 | 42.00 | 199 |
Two findings from that sweep worth carrying:
1. On this model the backend matters ~10× more than the tier. The three 4-bit tiers span ~1.5 %;
switching backend is worth 8.6–14.4 %. ⚠️ This is model-specific — on a different model we measured
the tier driving a 22 % swing while the backend barely moved. Measure your own.
2. Q6 buys nothing over Q8 on speed (42.0 vs 42.3 ROCm). Choose Q6 for footprint, not throughput.
Vulkan stability on gfx1151 — ⚠️ verified, and this model is clean
This model was tested directly for silent truncation and passed on every tier, on both backends.
12 reps × 4 workloads per arm, n_predict = 400:
| Scope | Generations | Truncated |
|---|---|---|
| This model — all six tiers on Vulkan | 288 | 0 |
| This model — COHERENT on ROCm | 48 | 0 |
| Sibling Ornith-1.5-9B-Abliterated — COHERENT on Vulkan | 48 | 8 — 17 % |
| Sibling Ornith-1.5-9B-Abliterated — COHERENT on ROCm | 48 | 0 |
Every tier here — COHERENT, FAST, STRIX_LEAN, Q6_AGENT, Q8_AGENT, Q8 — returned 0 / 48 on
Vulkan. The sibling, built from the same recipe on the same box with the same binary and driver, fails the
same test. The failure sits at the (9B model × Vulkan backend) intersection: it is not the tier name,
not the ROCmFP4 format, and not Vulkan on its own — this model runs Vulkan clean, across the whole ladder.
We still document it here because the two models share a recipe, a build, a driver and a box, and only one
of them fails. If you are running a different model on gfx1151, that is the case you should check.
What the sibling does — and why a median-only benchmark misses it
Across the sibling's full six-tier ladder (12 generations per arm, 72 per backend):
| Tier (9B sibling) | ROCm0 | Vulkan0 |
|---|---|---|
| Q4_0_ROCMFP4_COHERENT | 0 / 12 | 3 / 12 |
| Q8_0_ROCMFPX | 0 / 12 | 1 / 12 |
| FAST · STRIX_LEAN · Q6_AGENT · Q8_AGENT | 0 / 12 each | 0 / 12 each |
| Total | 0 / 72 | 4 / 72 |
It is tier-variable, not tier-exclusive — worst on COHERENT, but it also hit Q8_0_ROCMFPX, so a tier
passing once does not make it immune.
It fails silently. The server returns 1 token — empty content — with no error, no crash, a clean exit
code and /health still 200. A benchmark reporting only a median cannot see it: it surfaces as an absurd
tg = 1000000 tok/s (a 1-token / 0 ms decode) or as a plausible "fast" outlier inside a range.
Other Vulkan failures on the same hardware
| Model | What Vulkan did |
|---|---|
| Qwen3.8-27B | Speculative decoding (MTP) does not work on Vulkan |
| Qwen3.8-Flash-Next (~99 GiB) | vk::DeviceLostError — vk::Queue::submit: ErrorDeviceLost on every arm; the GPU is lost at queue submit and the server aborts before serving a token |
What to do about it
- Validate generation length, not just throughput. Send N completions and count how many reach your
n_predict. That check costs minutes and is the only thing that catches this class of bug.
- If you use speculative decoding, use ROCm. We have not seen spec-decode work on Vulkan on gfx1151.
- If a Vulkan run aborts with
ErrorDeviceLost, that is the driver, not the file — the same GGUF serves
on -dev ROCm0. One flag: no re-download, no rebuild.
- Across every model, tier and backend we measured on this hardware, **ROCm produced zero truncated
generations** (0 / 168). It is the conservative choice; on this model Vulkan is both clean and faster.
Provisional pattern, stated with its sample size. The large Vulkan decode win appeared on the MoE
model (this one) and not on the two dense models measured on the same box (≈ −0.5 % and +0.8 %). That
is one MoE against two dense — a hypothesis to test on your own model, not a rule to inherit.
Head protection — verified, not assumed
Left to defaults, several ftypes emit a 4-bit output.weight — we have measured that degrading a model
from 4/5 to 1/5 on held-out checks. This file was quantised with an explicit --output-tensor-type q6_K
and then read back and verified by exact tensor name:
output.weight = q6_K ✅
token_embd.weight = q6_K ✅
733 tensors · 40/40 blocks · expert_count 256 · expert_used 8⛔ ftype 105 (STRIX) is deliberately not published in this family — it leaves output.weight and 78
*_shexp tensors at 4 bit. This tier (102) is the coherent 4-bit build.
Source integrity — verified before quantising
A 35B MoE abliteration can ship a de-fused checkpoint whose expert tensors lost their expert
dimension: it loads, it scores badly, and it is easy to misread as "ablation does not survive MoE".
Gated before building:
base_model ornith-ai/Ornith-1.5-35B-A3B
base_revision e4dfb35a93d4b6822a811a7676f3488514abe7e2
method refusal-direction orthogonal weight projection
norm_preserve true
expert_down 25 physical tensors / 6,400 logical paths (25 × 256 experts)Independently confirmed from the BF16 header: 120 _exps + 160 _shexp, **0 expert tensors missing the
expert dimension**, blk.0.ffn_down_exps.weight = [512, 2048, 256]. The projection reached all 256 experts.
Reproduction block
binary ROCmFPX @ charlie12345/ROCmFPX, build-hipvk
(GGML_HIP=ON GGML_VULKAN=ON GGML_NATIVE=ON, gfx1151, Release,
CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang, LLAMA_CURL=OFF)
source PocketAiHub/Ornith-1.5-35B-A3B-Abliterated-GGUF :: Ornith-1.5-35B-A3B-Abliterated-BF16.gguf
69,376,637,024 bytes · 733 tensors · arch qwen35moe
quantize llama-quantize --output-tensor-type q6_K Q4_0_ROCMFP4_COHERENT 12
output 19,846,166,304 bytes · 4.58 BPW
host Ryzen AI Max+ 395 · Radeon 8060S gfx1151 · 128 GB unified · ROCm 7.2.4 · Ubuntu 24.04 · kernel 7.0.0-30
state box idle, 0 co-resident seats
method 4 fixed prompts · temp 0 · top_k 1 · 400 tok · cache_prompt:false · discard rep 1 · median of 3A number without its binary is a rumour. Every figure here came from the binary and argv above.
Perplexity — measured
wikitext-2-raw, 100 chunks, identical for every tier so they are directly comparable to each other.
-c 512, ROCm0, -ngl 999 -fa on.
| Tier | ftype | PPL | ± |
|---|---|---:|---:|
| Q6_0_ROCMFPX_AGENT | 114 | 8.0187 | 0.13410 |
| Q8_0_ROCMFPX_AGENT | 115 | 8.0805 | 0.13536 |
| Q8_0_ROCMFPX | 111 | 8.1875 | 0.13828 |
| Q4_0_ROCMFP4_COHERENT | 102 | 8.2110 | 0.13810 |
| Q4_0_ROCMFP4_FAST | 103 | 8.2837 | 0.13864 |
| Q4_0_ROCMFP4_STRIX_LEAN | 106 | 8.3226 | 0.14000 |
⚠️ Read the error bars before ranking these. The three 4-bit tiers span 0.11 PPL against ±0.138
uncertainty — they are statistically indistinguishable at this sample size. The ordering shown is
suggestive, not significant. What is supported: the 6/8-bit tiers sit measurably below the 4-bit group,
and even there Q6 vs COHERENT (0.192 apart) is only borderline.
⚠️ This is a 100-chunk measurement, not full wikitext-2. It is valid for comparing these six files to
each other. Do not compare these figures against a full-corpus PPL from another card.
⭐ Note Q6 (114) scoring below both Q8 tiers — with overlapping error bars, treat that as "the 6/8-bit
tiers are equivalent", not as evidence Q6 is better than Q8.
Verification — every tier, measured
| Tier | loads ROCm0 | loads Vulkan0 | coherent output | tool calling | mmproj loads |
|---|:--:|:--:|:--:|:--:|:--:|
| Q4_0_ROCMFP4_COHERENT | ✅ | ✅ | ✅ | 3/3 | ✅ |
| Q4_0_ROCMFP4_FAST | ✅ | ✅ | ✅ | 3/3 | ✅ |
| Q4_0_ROCMFP4_STRIX_LEAN | ✅ | ✅ | ✅ | 3/3 | ✅ |
| Q6_0_ROCMFPX_AGENT | ✅ | ✅ | ✅ | 3/3 | ✅ |
| Q8_0_ROCMFPX_AGENT | ✅ | ✅ | ✅ | 3/3 | ✅ |
| Q8_0_ROCMFPX | ✅ | ✅ | ✅ | 3/3 | ✅ |
12/12 backend load arms passed. 0 failures.
Coherence — actual generated output, identical in structure across all six tiers (prompt: *"Write a
Python function to merge two sorted lists. Code only."*):
def merge_sorted_lists(list1, list2):
merged = []
i = j = 0
while i < len(list1) and j < len(list2):
if list1[i] <= list2[j]:
merged.append(list1[i])Correct two-pointer merge with proper bounds. Read, not assumed — a fast gibberish generator is a failure
we have shipped before.
Tool calling — 3 distinct calls per tier, each verified that the argument round-tripped correctly
(city name present in the emitted tool_calls), not merely that some tool call was returned.
⚠️ Vision: the mmproj loads on all six tiers — image-grounded accuracy is NOT scored. "Loads" and
"works" are different claims and we are only making the first one.
Not yet measured
- Image-grounded vision accuracy (the mmproj loads; output not scored against ground truth).
- Context beyond 32768. MTP/speculative decoding (source ships
mtp_included: false). - Full-corpus perplexity (the figures above are a 100-chunk run).
License and attribution
- Base model:
ornith-ai/Ornith-1.5-35B-A3B@e4dfb35a - Abliteration: by huihui-ai —
Huihui-Ornith-1.5-35B-A3B-abliterated - BF16 GGUF + manifests:
PocketAiHub— whose
abliteration-manifest.json and conversion-validation.json made the integrity gate above possible
- This repository contributes only the ROCmFP4 quantisation, the dual-backend measurements, and the
verification. All model credit belongs upstream.
Acknowledgements
huihui-ai for the abliteration · PocketAiHub for the BF16 and — unusually and valuably — for
shipping manifests that make source integrity checkable rather than assumed · ornith-ai for
Ornith-1.5-35B-A3B · the ROCmFPX project for the FP4/FPX types and Strix Halo kernels.