OxCoder-9B — ROCmFP4 imatrix for AMD Strix Halo (gfx1151)

Importance-matrix-calibrated ROCmFP4 quantizations of OrionLLM/OxCoder-9B (9,409,813,744 parameters, 32-layer dense Qwen3.5 hybrid: 24 Gated DeltaNet linear-attention and 8 full-attention layers, every 4th being full attention; hidden 4096, FFN 12288, 16 attention / 4 KV heads, 262,144-token context, text + image). Companion to the standard build kingjones777/OxCoder-9B-MTP-ROCmFP4-GGUF — the same three 4-bit tiers, same grafted MTP head, same vision projector, same prompt-cache patch; the only difference in the weights is how each 4-bit block's scale was chosen.

What the imatrix changes

ROCmFP4 has an importance-weighted quantizer path: with --imatrix, each block's scale is chosen by an exhaustive search that minimises error weighted by how strongly the calibration activations use each weight, instead of the unweighted default. It changes which scales are picked at the same bit width and tensor types — so it moves quality, not size, and per-token compute is identical.

Measured effect

Quality is graded against the BF16 GGUF (reference logits computed on CPU) on a held-out corpus (wikitext-2 test, -c 2048, 40 chunks × 1,023 scored tokens each — the second half of every window, less its first token — = 40,920), never on the imatrix calibration text. KLD is the per-token KL divergence of each quant's next-token distribution from BF16's on the same tokens — far more sensitive than perplexity. The calibration text and the grading text are different corpora.

σ = difference divided by the two runs' combined standard error. The two runs score the same tokens, so this is conservative (paired noise is smaller).

The imatrix measurably improves all three tiers. STRIX_LEAN improves the standard file on mean KLD (-30.6 %, 6.2σ; 0.0879 → 0.0610); median -32.8 %, 99th-pct -32.2 %, top-1 +1.46 pp; COHERENT improves the standard file on mean KLD (-35.9 %, 7.4σ; 0.0857 → 0.0550); median -36.0 %, 99th-pct -34.1 %, top-1 +1.74 pp; FAST improves the standard file on mean KLD (-28.4 %, 5.8σ; 0.0890 → 0.0637); median -33.5 %, 99th-pct -31.8 %, top-1 +1.50 pp.

Where the quality numbers come from. Measured directly on these files, against BF16 logits computed on CPU in the same session. The BF16 model's own first-window perplexity was CPU 6.6290, Vulkan0 6.6346 (+0.08 %), ROCm0 6.6463 (+0.26 %): the two GPU backends disagree with CPU and with each other, so the reference logits come from pure CPU (-dev none -ngl 0 --no-op-offload), and that run's own first-window value (6.6290) equals the CPU control's. Every file was graded on ROCm0 (the columns above); the standard STRIX_LEAN file was also graded on Vulkan0. The standard STRIX_LEAN grade was run twice on ROCm0 and all 40 per-chunk rows matched.

Same file, same reference, graded on each backend — standard STRIX_LEAN KLD ROCm0 / Vulkan0: 0.0879 / 0.0886 (+0.9 %, 0.2σ). The two backends agree within noise on this file.

Which file should I use?

Start with STRIX_LEAN. COHERENT's KLD is within noise of it (-9.9 %, 1.6σ) for 315 MiB more; STRIX_LEAN decodes within 0.4 % of COHERENT on ROCm0 (59.03 vs 59.28 tok/s; per-run ranges 57.78–59.12 and 58.16–59.53) and within 7.3 % of it on Vulkan0 (59.18 vs 63.83), and prefills 10.4 % faster than it on ROCm0 (897.6 vs 812.9 tok/s).

FAST does not buy a clear speed gain here: FAST decodes within 1.7 % of STRIX_LEAN on ROCm0 (60.01 vs 59.03 tok/s; per-run ranges 58.76–60.11 and 57.78–59.12) and within 3.3 % of it on Vulkan0 (61.13 vs 59.18), and prefills within 1.4 % of it on ROCm0 (910.3 vs 897.6 tok/s); its KLD is within noise of STRIX_LEAN's (0.0637 vs 0.0610, +4.5 %, 0.7σ).

KLD order (lower is closer to BF16; KLD gaps under 2σ are called within noise): COHERENT 0.0550, STRIX_LEAN 0.0610 (+10.9 % vs COHERENT), FAST 0.0637 (+15.9 % vs COHERENT). STRIX_LEAN is 315 MiB smaller than COHERENT; FAST is 178 MiB smaller than STRIX_LEAN. Speed gaps are called a tie below 10.3 % for decode and below 3.0 % (ROCm0) / 3.0 % (Vulkan0) for prefill (3.0 % when the two backends are compared): the larger of 3 % and the widest gap measured between two files that run identical tensors — each imatrix file and its standard twin at the MTP knee (decode 10.3 %; prefill 1.3 % on ROCm0, 0.2 % on Vulkan0). The decode floor is the widest because each twin's drafts are accepted at a different rate (STRIX_LEAN on ROCm0: 0.750 standard vs 0.641 imatrix): the imatrix changes the main model's weights, so the head sees different hidden states and is checked against different target tokens.

For scale: the imatrix STRIX_LEAN's KLD is lower than the standard COHERENT's (0.0610 vs 0.0857, -28.9 %, 5.7σ), STRIX_LEAN is 315 MiB smaller than COHERENT.

TG = decode tokens/s after the 7,094–7,102-token code prompt, grafted MTP head, --spec-draft-n-max 3. PP = prefill tokens/s on ROCm0. ¹ The BF16 PPL shown is the paired base every "×" ratio is computed against (averaged over the same scored tokens in the KL-divergence runs). The standalone BF16 run's own summary line reads 6.6532 ± 0.0813. ² Quality columns: see Where the quality numbers come from above. ³ BF16 conversion of the grafted checkpoint (MTP block included); not published. ⁴ BPW as printed by llama-quantize: bits per weight over the 9,197,093,888 weights in each GGUF — the checkpoint's 8,953,803,264 text weights plus the grafted 243,290,624-weight MTP block. The 9,409,813,744-parameter count above is the checkpoint's: it includes the 456,010,480-weight vision tower, which ships in the projector file, and no MTP block.

The imatrix changes scale values, not tensor types or sizes: tensor by tensor, each imatrix file has the same names, types and byte sizes as its standard twin (the files differ in size by 256 / 256 / 256 bytes, all of it in the GGUF header, which carries 4 extra keys in each imatrix file: quantize.imatrix.chunks_count, quantize.imatrix.dataset, quantize.imatrix.entries_count, quantize.imatrix.file), so every forward pass does the same work. With the MTP head, decode speed also depends on how many drafts are accepted, and that differs between twins — imatrix vs standard, at the MTP knee: STRIX_LEAN on ROCm0 59.03 vs 65.80 tok/s (acceptance 0.641 vs 0.750); STRIX_LEAN on Vulkan0 59.18 vs 60.98‡ tok/s (acceptance 0.639 vs 0.672); COHERENT on ROCm0 59.28 vs 57.40† tok/s (acceptance 0.665 vs 0.628); COHERENT on Vulkan0 63.83 vs 60.53‡ tok/s (acceptance 0.737 vs 0.673); FAST on ROCm0 60.01 vs 64.81† tok/s (acceptance 0.656 vs 0.738); FAST on Vulkan0 61.13 vs 61.72 tok/s (acceptance 0.659 vs 0.672). The largest of these decode gaps is 10.3 %; both cards call decode gaps below 10.3 % a tie. † / ‡: standard rows whose first measurement overlapped an unrelated job on the box (standard card, Vulkan0 prefill and CPU activity).

Full speed tables (depth sweep, DFlash, both backends, code vs prose), the grafted-head provenance, prompt-cache, tool-calling and vision results are on the standard card.

Quick start

1. Download

hf download kingjones777/OxCoder-9B-MTP-ROCmFP4-imatrix-GGUF --local-dir ~/models/oxcoder

2. Build llama-server with the prompt-cache patch — ROCmFPX at the measured commit (ROCm and Vulkan prerequisites: the project's build guide)

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout d3ca53726109b46fd5cd6fa42f688c707079b44e
git apply ~/models/oxcoder/recipe/patches/mtp-prompt-cache-fix.patch
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

(The CMake options of the measured build are listed in Reproduction.)

d3ca537 is also in the history of the official ROCmFPX/ROCmFPX repository; the patch was verified against d3ca53726109b46fd5cd6fa42f688c707079b44e only.

3. Serve — the MTP head is inside the model file, so there is no --model-draft:

Text and tool calls, with MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/oxcoder/OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.0 \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

Images (--mmproj), without MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/oxcoder/OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj ~/models/oxcoder/mmproj-OxCoder-9B-BF16.gguf \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

⛔ Images: with the MTP head loaded, image requests fail on this build (measured). For image input, drop the two --spec-* lines.

(LD_LIBRARY_PATH avoids a soname clash on machines that also have a Vulkan-only llama.cpp build.) The exact measured argv is in Reproduction.

4. Call — thinking off (OxCoder's SWE-bench sampling: temperature 1.0, top_p 0.95):

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

⚠️ On an unpatched build the in-file head still drafts, but prompt caching turns off while it is loaded — every turn re-prefills the whole conversation (measured). For multi-turn / agentic use there, drop the two --spec-* lines.

Requires a llama.cpp build with ROCmFP4 / ROCmFPX tensor-type support; stock llama.cpp rejects these tensor types.

Reasoning controls

OxCoder ships its own chat template (template_version qwen3.6-froggeric-v21.3, in the source repo's chat_template.jinja). It reads these chat_template_kwargs:

A <|think_off|> or <|think_on|> tag inside a system or user message overrides enable_thinking (the last tag wins) and is removed from the text.

A short tool result that starts like an error (error:, Traceback, fatal:, command not found …) gets a "⚠️ SYSTEM WARNING" appended by the template; after two such results in a row (no user turn in between), the next generation prompt also gets an empty think block, regardless of enable_thinking. A developer message is treated as system; any other unknown role is rendered as a user turn [role]: content.

With thinking on and a small max_tokens, the whole budget can go to reasoning and content comes back empty — raise max_tokens before concluding the model is broken.

Tool calls in the default format are parsed by llama.cpp natively through the Jinja chat template (on by default). OxCoder's own card reports SWE-bench at temperature 1.0 / top_p 0.95 and Terminal-Bench / NL2Repo at temperature 1.0 / top_p 1.0. The tools gate reported on the standard card used temperature 1.0, top_p 0.95, top_k disabled, max_tokens 8192.

Quantization methodology

llama-imatrix -m OxCoder-9B-MTP-BF16.gguf -f calibration_datav3.txt -o OxCoder-9B.imatrix \
  -c 512 -b 512 -dev none -ngl 0 --no-op-offload -t 16
llama-quantize --imatrix OxCoder-9B.imatrix --output-tensor-type q6_K \
  OxCoder-9B-MTP-BF16.gguf OxCoder-9B-MTP-imatrix-Q4_0_ROCMFP4_STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEAN 16
llama-quantize --imatrix OxCoder-9B.imatrix --output-tensor-type q6_K --token-embedding-type q6_K \
  OxCoder-9B-MTP-BF16.gguf OxCoder-9B-MTP-imatrix-Q4_0_ROCMFP4_COHERENT.gguf Q4_0_ROCMFP4_COHERENT 16
llama-quantize --imatrix OxCoder-9B.imatrix --output-tensor-type q6_K \
  OxCoder-9B-MTP-BF16.gguf OxCoder-9B-MTP-imatrix-Q4_0_ROCMFP4_FAST.gguf Q4_0_ROCMFP4_FAST 16

Receipts that the weighted path was actually taken, and that each shipped file differs from its standard twin:

Reproduction

server   : https://github.com/charlie12345/ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e
           + recipe/patches/mtp-prompt-cache-fix.patch (sha256 412a0173f60faa937018f0153efe9b5d9230c1606735502485e7e520fffb68dd)
           built target llama-server with the same CMake options as below; sha256 14e9f1299240c891478c820b8c20811ca5721044d2e63e23799eaa9be26b30bd
tools    : same commit, unpatched (quantize / imatrix / perplexity; the patch touches only common/ and tools/server/)
           build dir /opt/llama-rocm/rocmfpx-724/build-hipvk, Release, Unix Makefiles, GGML_HIP=ON GGML_VULKAN=ON
           GGML_HIP_GRAPHS=ON GGML_HIP_NO_VMM=ON GGML_NATIVE=ON AMDGPU_TARGETS=gfx1151 LLAMA_CURL=OFF
           (this CMake forwards AMDGPU_TARGETS to GPU_TARGETS, the spelling used in Quick start)
           CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang
           sha256 llama-quantize   1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
           sha256 llama-imatrix    bfdce5c21a01bdc8043d69335b74a7b68b8335ae989991a1e10e21d5faeaeb8e
           sha256 llama-perplexity c2175925bb0c61df89ff787845d8e03a35403f684be52719ac2987b8ca636a53
           sha256 llama-server     d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7   (unpatched; used for the unpatched control rows: prompt cache and vision)
source   : OrionLLM/OxCoder-9B  revision 07ba04e2149bace1b43c66448471a70378a7a708
model    : OxCoder-9B-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf (the argv below; every file was measured the same way)
argv     : /opt/llama-rocm/rocmfpx-724-mtpcache/build-hipvk/bin/llama-server -m /mnt/models/oxcoder-9b/out-imat/OxCoder-9B-MTP-imatrix-Q4_0_ROCMFP4_STRIX_LEAN.gguf -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -c 65536 -b 2048 -ub 1024 --host 127.0.0.1 --port 18600 --no-webui --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.0
env      : LD_LIBRARY_PATH=<patched build>/bin:/opt/rocm-7.2.4/lib
           HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
box      : aimax — AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (gfx1151), 124 GiB RAM, GTT 131072 MiB,
           kernel 6.17.6-061706-generic, ROCm 7.2.4
protocol : 7,094–7,102-token code prompt, 256 generated tokens, temp 0 / top_k 1, ignore_eos, cache_prompt false,
           1 warm-up + median of 3, no co-resident models (box iced)
measured : 2026-09-16 to 2026-09-17, by the pipeline in recipe/ (every raw number in recipe/results_summary.json and recipe/raw/)

Files

Known issues and limits

• Calibration text is general-purpose English/code (bartowski calibration_datav3.txt); an imatrix built from your own domain can do better on that domain.
• Graded on wikitext-2 test, a different corpus from the calibration text. Improvements on reasoning/code tasks were not measured.
• Quantized without importance weights (the O3 llama-quantize logs print did not find weights for each): the output projection output.weight (llama-imatrix skips it unless --process-output is given); the token embedding token_embd.weight (a table lookup, not a matrix multiply); the 8 tensors of the grafted MTP block blk.32.* (the calibration forward pass does not run it).
• The MTP head is grafted from the base model. The 15 mtp.* tensors come from Qwen/Qwen3.5-9B at revision c20223623576 (recipe/graft/GRAFT_PROVENANCE.json). They were trained for that checkpoint, not for OxCoder's fine-tune; their usefulness here is the measured acceptance and speed on this box, not a task-accuracy claim.
• MTP + image requests fail on this build (they work with the head unloaded) — see Vision. For image input, drop the two --spec-* lines.
• Prompt caching with MTP needs the included server patch. On the unpatched server the in-file head still drafts (40 of 62 drafted tokens accepted in that control), but the measured second turn reused 0 of its 7,067 prompt tokens; for multi-turn use on an unpatched build, drop the two --spec-* lines.
• preserve_thinking defaults to true in this chat template: earlier assistant turns are re-rendered with their reasoning, so multi-turn contexts grow faster than with templates that drop it. Pass "preserve_thinking": false in chat_template_kwargs to drop it.
• Tool calling was measured only with the template's default tool_call_format (XML parameters); the JSON variant ("tool_call_format": "json") was not tested.
• llama-server's host-RAM prompt cache defaults to 8 GiB (-cram 8192). On a shared box, set -cram explicitly.
• Measured on Linux only (AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, ROCm 7.2.4, patched d3ca537).
• Not measured: decode beyond a 7,458-token prompt (a 262,144-token context was loaded in the memory test, not benchmarked), long-context quality, video input, concurrency above 1, and task-level accuracy. Perplexity/KLD measure next-token fidelity to BF16 on prose, not reasoning or code correctness. OxCoder's own SWE-bench / Terminal-Bench / NL2Repo numbers are for the unquantized checkpoint, not these files.

License and attribution

Apache-2.0, inherited from the base model. Weights: OrionLLM (OrionLLM/OxCoder-9B). MTP head weights: Qwen (Qwen/Qwen3.5-9B, Apache-2.0). Calibration text: bartowski's calibration_datav3. ROCmFP4 / ROCmFPX: the ROCmFPX project. Patch, graft, imatrix, quantization and measurements: kingjones777.