Orca Cyber 27B, XE2MX4

Unrotated XE2MX4 requant of the Orca Cyber IQ4_XS GGUF, built for Intel Xe2 dGPUs through the SYCL backend in llama.cpp-exl3xe2cuda.

This is a custom GGUF type, stock llama.cpp does not load it, CUDA and Vulkan do not run it. It is not EXL3 or MXFP4, each block stores 32 signed 4-bit weights and one FP16 scale, 4.5 bits per weight. Output and token embeddings stay Q8_0, embedded MTP is retained.

The XE2MX4 loader and quantizer are now on the GitHub fork's main branch, use that fork rather than stock llama.cpp.

Quality

Held-out wikitext-2 test, 100 chunks of 512 tokens, lower PPL is better:

S48R is what we used locally before this file, we switched to unrotated because PPL was better at similar speed. This is a second quantization from IQ4_XS, not a BF16 comparison, the PPL values are prior measurements on the same test text and do not guarantee identical results on every task.

Measured on Arc Pro B65

llama-server HTTP, patched oneDNN, 262144 context, q4_0 target/draft KV, cheap MTP p-min 0, batch/ubatch 3072:

• 26k prompt, 256 output tokens: PP 2034.1 tok/s, TG 38.23 tok/s, drafts accepted 102/153, reading questions 8/8.
• Single planted-code recall passed at six depths, up to 250736 prompt tokens. At that depth PP was 760.7 tok/s, TG 27.8 tok/s over a 65-token answer. This is a recall smoke test, not a general long-document accuracy benchmark.
• Eight simultaneous requests, 16384 context per request, repeated 2343-token prompt and forced 4096-token output each: total throughput 89.94 tok/s including prefill, median request TG 13.31 tok/s, peak VRAM 26.14 GiB. Four requests reached 99.11 total tok/s, scaling was uneven. This is concurrent inference, not an agent-quality test, and the repeated prompt favors MTP.

Other Xe2 dGPUs are architecture targets, not benchmarked cards.

Run

Build the XE2MX4-enabled SYCL fork and the patched oneDNN described in the runtime instructions, the instructions are available on the main branch. Put the patched oneDNN lib directory first in LD_LIBRARY_PATH.

ONEAPI_DEVICE_SELECTOR=level_zero:0 ./build-sycl/bin/llama-server \
  --device SYCL0 -m OrcaSAQ-2-27B-Uncensored-XE2MX4.gguf -ngl 99 \
  --flash-attn on --ctx-size 262144 -ctk q4_0 -ctv q4_0 \
  -b 3072 -ub 3072 --spec-type draft-mtp --spec-mtp-cheap \
  --spec-draft-p-min 0 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
  --reasoning on --reasoning-effort low --jinja

Select the GPU you want to use, the example exposes device 0. Keep reasoning on for agent/coding use, the throughput tests above disabled thinking per request.

Memory

The weights remain on the GPU for decode, RAM offload is an additional attention/projection pack, not the whole model or CPU expert offload. With host-pack enabled that pack lives in system RAM and is uploaded for prefill, saving resident VRAM at the cost of host memory and transfer overhead.

File size is not runtime VRAM, KV cache, recurrent state, compute buffers and temporary packs add memory, especially with longer contexts and more slots. Peak minus idle is not just the host pack. The fast path needs patched oneDNN, stock oneDNN uses a slower fallback.

Source and license

Source model: orcarouter/OrcaSAQ-2-Cyber-27B-Uncensored-GGUF. Model license is Apache-2.0. This repository supplies the requantized GGUF, it does not claim authorship of the source model.