Sarvam-30B — ROCmFP4 for AMD Strix Halo (gfx1151)

Four ROCmFP4/ROCmFPX tiers of sarvamai/sarvam-30b, built from the fp32 release and measured on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151, 128 GB unified, ROCm 7.2.4). Sarvam-30B is a 128-expert MoE (top-6, shared expert, 19 layers, rope_theta 8e6, 262 144 vocab) with strong Indic-language coverage and native tool calling.

Every number on this card was measured on these exact files. Nothing is estimated.

Which file should I use?

Start with FAST. On this model it is the fastest tier on both decode and prefill, and all four score identically on the capability checks below — there is no quality reason to pay for the larger files unless you specifically want 8-bit.

Quick start — the backend is one flag

# HIP / ROCm — best prefill (long prompts, RAG, cold starts)
llama-server -m Sarvam-30B-Q4_0_ROCMFP4_FAST.gguf \
  -dev ROCm0 -fa on -ngl 999 --no-mmap -fit off -np 1 \
  -b 2048 -ub 1024 -t 16 --poll 100 -c 8192 --jinja

# Vulkan — SAME FILE. Only -dev changes.
llama-server -m Sarvam-30B-Q4_0_ROCMFP4_FAST.gguf -dev Vulkan0 ...

⛔ You need a ROCmFPX build — stock llama.cpp will NOT load these

These use the ROCmFP4/ROCmFPX tensor types (ggml types 100-119), which upstream llama.cpp does not have. Build one of:

# official — recommended, and what these files were verified against
git clone https://github.com/ROCmFPX/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGGML_VULKAN=ON -DAMDGPU_TARGETS=gfx1151 \
      -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

The model carries general.architecture = bailingmoe2, which the official repo supports directly — see the conversion note below.

The conversion — why this is bailingmoe2

sarvamai/sarvam-30b declares SarvamMoEForCausalLM / model_type: sarvam_moe, and no runtime implements that architecture. It does not need one: Sarvam-MoE shares the BailingMoeV2 graph, differing only in full rotary (no partial_rotary_factor) and a zero-mean-normalised expert bias. The converter maps it to MODEL_ARCH.BAILINGMOE2 and writes those two adjustments into the GGUF, so these files load on any ROCmFPX build with bailingmoe2 — including the official repo, unmodified.

Measured performance — all four tiers × both backends

Median of 3 timed generations after a discarded warm-up, temperature 0, -c 8192, single slot. Prefill measured with a ~7 800-token prompt, cache disabled.

The findings that matter

• HIP wins prefill decisively — by 42-51% (1696 vs 1167 on FAST). If your workload is agentic or RAG, where every turn re-prefills a growing conversation, use -dev ROCm0.
• Decode is close, and HIP wins three tiers of four. This is the opposite of what we measured on Ornith-1.5, where Vulkan took decode. Backend ranking does not transfer between models — measure it.
• The 8-bit tier costs 33% of decode (54.4 vs 81.6) for no measured capability gain. It is here for people who want 8-bit weights, not because it scores better.

Silent-truncation gate — 32 generations, 0 truncated

Every tier × backend cell ran 4 generations checked for the 1-token silent stop. 0/4 in all eight cells. Facts and tool calling passed in all eight.

⛔ Serving note — this model does not stop thinking

Sarvam-30B is a reasoner, and its reasoning length is highly non-deterministic even at temperature 0. The same coding prompt, same build, same 4 000-token budget:

This is upstream behaviour, not damage from the quantisation. Sarvam's own sarvamai/sarvam-30b-gguf Q4_K_M does exactly the same thing on the same prompts — see the control arm below.

Practical consequences:

• Give it room. Short prompts and tool calls are fine at 1 500 tokens. Long-form generation may need 4 000+, and may still not emerge on a given draw. A small budget makes this model look broken when it is not — you will get HTTP 200 with an empty content and all the tokens in reasoning.
• chat_template_kwargs {"enable_thinking": false} is IGNORED by this model — it still emits 5 500-6 000 characters of reasoning. The switch that works on Qwen3.8 does not work here.
• For deterministic long-form output, retry the draw rather than raising the budget indefinitely.

Head protection — verified, not assumed

Left to defaults, several ftypes emit a 4-bit output.weight. hidden_size is 4096 here (divisible by 256), so a K-quant head is legal, and every file was quantised with an explicit --output-tensor-type and then read back by exact tensor name to confirm it applied:

Capability control arm — against the vendor's own quant

The question "is our quantisation damaging this model?" is only answerable against a reference, so we downloaded sarvamai/sarvam-30b-gguf (Q4_K_M) and ran identical prompts and budgets on the same machine and binary:

Identical behaviour, ~13% faster decode. The over-thinking is the model; the quantisation is faithful.

Reproduction block

model     : sarvamai/sarvam-30b  (fp32, 26 safetensors, 128.6 GB)
            config: sarvam_moe / SarvamMoEForCausalLM, 19 layers, hidden 4096,
            128 experts top-6, moe_intermediate 1024, rope_theta 8e6, vocab 262144,
            tie_word_embeddings = false
convert   : rocmfpx-fork @ e771235   convert_hf_to_gguf.py --outtype bf16
            -> Sarvam-30B-BF16.gguf, 264 tensors, 64.3 GB, arch bailingmoe2
quantize  : llama-quantize sha256 0acbee2969bf6cad…
            --output-tensor-type q6_K  <in> <out> Q4_0_ROCMFP4_COHERENT   16
            --output-tensor-type q6_K  <in> <out> Q4_0_ROCMFP4_STRIX_LEAN 16
            --output-tensor-type q6_K  <in> <out> Q4_0_ROCMFP4_FAST       16
            --output-tensor-type q8_0  <in> <out> Q8_0_ROCMFPX_AGENT      16
serve bin : rocmfpx-fork build-hipvk, llama-server sha256 e860b763d5e8f496…
            (HIP + Vulkan in one binary; -dev selects)
verified  : also loads and serves on official ROCmFPX/ROCmFPX @ 8124bde
box       : Ryzen AI Max+ 395 / Radeon 8060S (gfx1151) / 128 GB unified
            Ubuntu 24.04, ROCm 7.2.4.  HSA_OVERRIDE_GFX_VERSION not set.
argv      : -dev {ROCm0|Vulkan0} -fa on -ngl 999 --no-mmap -fit off -np 1
            -b 2048 -ub 1024 -t 16 --poll 100 -c 8192 --jinja
measured  : 2026-09-01

SHA256

fcdc2121dbc34a85b2f1cc763653a4738d3dd4c0417391a7304ea85dd0a3cad2  Sarvam-30B-Q4_0_ROCMFP4_COHERENT.gguf
53884cb8aa27175a1c0b2e7eb9567f176c52c468bad93644e57b8e63de72518b  Sarvam-30B-Q4_0_ROCMFP4_FAST.gguf
a2cb2c3315facb42c85a164b495f210ab919724991f7d3af234dd2670eff7276  Sarvam-30B-Q4_0_ROCMFP4_STRIX_LEAN.gguf
b5ae84eaed973e381171f7089e26563b7d36b1eefaef6e0e1698586e62c74073  Sarvam-30B-Q8_0_ROCMFPX_AGENT.gguf

Credits

Sarvam AI — the base model and its Apache-2.0 licence. ROCmFP4/ROCmFPX quantisation types are the work of ROCmFPX (Carlo Pasquale).