Escha Qwen3.8-27B W2 GGUF (dense)

I ported EschaLabs/Qwen3.8-27B-Escha-W2 to llama.cpp. Weights are theirs. This is not a requant.

I decode their native 2-bit escha code in-kernel, through a new op, GGML_OP_ESCHA_MUL_MAT. The quantized payload is the same bytes as the safetensors. Nothing is unpacked to a dense tensor on disk or in VRAM.

This is the dense 27B. My earlier port of their MoE 35B is a separate repo and a separate branch.

Stock llama.cpp cannot load these files. You need my fork, branch escha-w2-dense:

https://github.com/Ajay9o9/llama.cpp-escha/tree/escha-w2-dense

If you only download the GGUF, it will not run.

Files

Two builds. Same weights, same kernel. They differ only in how the token embedding and the output head are stored.

Both are 2054 tensors, 2.469 bits per weight on the quantized tensors, K=2 and K=3 mixed. 64 layers, 48 linear attention and 16 full attention (full_attention_interval 4), 5120 hidden, 24 heads over 4 KV heads, head dim 256, vocab 248,320.

The Q8_0 build is a repack of the source int8, not a requantization. The row's f16 scale goes into every block and the payload is copied verbatim. I dequantized 64 rows out of the GGUF and compared against int8 * scale from the safetensors: scales identical per row, payload identical, max abs diff 0.0. 2052 of the 2054 tensors are byte-identical between the two files.

There is also an MTP draft model, for speculative decoding:

It is optional and it pairs with either build above. See MTP speculative decoding.

Which one to take

Take Q8_0. It is 2.38 GB smaller and 3.4% faster to generate.

The Q8_0 file has exact weights where F16 rounds them by about 0.02%, so I expected it to land closer to Escha's runtime. It lands slightly further: 318/320 against 319/320 on top-1. llama.cpp's Q8_0 matmul (vec_dot_q8_0_q8_1) quantizes the activations to 8 bits for an integer dot product, while the F16 path leaves them in fp32. So the trade is "weights rounded, activations exact" against "weights exact, activations quantized", and the second error is larger. That is read off the kernel and the measurement. I did not instrument it.

One position in 320 is inside the noise. Not a reason to carry 2.38 GB more.

Quality vs Escha SGLang

Same GPU, same day. I send token ids, not text, so both stacks see the same context and tokenizer differences cannot leak in. Teacher-forced: the prefix is fixed at every position, so a miss at i-1 does not poison i.

No drift as context grows. Mean JS divergence over the top-20 was 0.0000 bits. Median absolute logprob difference 0.006 (F16) and 0.013 (Q8_0).

My fp32 build and my fp16 tensor-core prefill build agree with each other on all 320 positions. That is what let me ship the tensor-core path. It separates "is the kernel correct" from "is fp16 good enough", which perplexity alone cannot do.

Op test against a numpy dense-fold reference, CPU and CUDA scored separately: rel RMS around 1e-6 at K=2 and K=3.

Perplexity: 7.4016 from a 512-context back-half estimator. EschaLabs report 7.43 from the same kind of estimator, so I am in the right place. I could not reproduce their 7.2652, no wikitext-2 on this machine. The logit agreement above is the evidence here, not this number.

Speed

One RTX 3090, power limit 250 W (stock is 350 W), single stream, batch 1, full GPU offload. This card is about 43% of a 4090 on FP32 and power capped on top of that.

The two builds prefill the same. That 13 tok/s gap is inside run to run spread on this card, so do not read anything into it.

Served, through llama-server at -c 70000, measured with llama-benchy, F16 build:

Decode drops up to 9% once the card is warm. I measured the same build at 23.12, then 22.60, then 20.99 in one sitting with nothing changed. The numbers above are from a cool card, so treat them as a best case rather than what you get an hour in. Deep context behaves differently: at 64k I got the same number hot or cold, because decode there is waiting on the KV cache and not on clocks.

EschaLabs' own SGLang runtime on this same 3090 gives 981 / 987 / 933 prefill and 31.17 / 29.05 / 28.37 decode at 8k / 16k / 32k. Their runtime is still faster at generation.

Prefill went from 212.4 to 708.1 tok/s over the port, 3.3x. Most of it was tensor cores (mma.m16n8k16, fp32 accumulate), cp.async staging, and hoisting integer division out of the tile loops. Decode went from 16.6 to 23.12.

The latest fork commit, c22479837 (escha: cut q/k re-reads in the gated delta net token loop), is the gated-delta-net piece of that. The old kernel gave each warp one column of the recurrent state, so every warp re-read the whole q and k vector. It now gives each warp four columns, reuses those loads, and prefetches the next token while the current one is still reducing. Same binary, GGML_CUDA_GDN_COLS=1 (old behaviour) against the default:

Decode does not move. GDN is about 19% of prefill, so a 31% faster GDN kernel shows up as 6% end to end. You need this commit or later of escha-w2-dense. The tables above are from it.

MTP speculative decoding

EschaLabs published an MTP head for this checkpoint on 2026-09-03, alongside their SGLang runtime 1.2.1. It was not there when I started this port, which is why earlier versions of this card said there was no MTP. I converted it so llama.cpp can use it too. Decode goes up 1.4x to 1.8x.

One layer (mtp_num_hidden_layers: 1), which is 0.86 GB of weights by itself. It sets mtp_use_dedicated_embeddings: false, so it reuses the main model's token embedding and output head instead of carrying its own. llama.cpp wants those in the draft file, so I put them there at Q8_0 and Q4_K. That is the 2.93 GB.

llama-benchy, same 3090 at 250 W, -c 65536, q8_0 KV, F16 main build, --spec-draft-n-max 3:

Draft acceptance is about 0.69 at 8k and 0.59 deeper, mean accepted length 3.07 down to 2.77 of 3 drafted. Single runs, no repeats, which is why 32k reads lower than 64k. Under a few percent is noise.

MTP on against MTP off. Same binary, same day, server defaults rather than the flags above, averaged over repeats:

So MTP is worth about 1.4x even on defaults, and the flags below take that to roughly 1.8x. Prefill on this build is the same either way inside the error bars.

For reference on the same card, EschaLabs' SGLang with their own speculative decoding gets 48.0 / 59.8 / 46.2 at 8k / 16k / 32k. Their runtime is still ahead, as it is without MTP.

A verify batch is cheap because the fork decodes the 2-bit weights once and reuses them across the batch instead of running the batch-1 path per token. Measured with llama-batched-bench, a 4-row verify step costs 1.50x a 1-row decode step, not 4x.

Run it with MTP

./build/bin/llama-server \
  -m  Escha-Qwen3.8-27B-W2-Q8E.gguf \
  -md Escha-Qwen3.8-27B-W2-MTP-F16-headQ4.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -c 65536 -np 1 \
  -b 2048 -ub 2048 \
  --temp 0 --top-k 1 \
  --host 127.0.0.1 --port 8080 --jinja

Set these four flags. They take MTP from about 1.4x to about 1.8x. Leave them off and you still get 1.4x:

• -np 1. Otherwise the server picks 4 slots and splits the batch.
• --temp 0 --top-k 1. Greedy. At the GGUF default of temp 1.0 the model accepts fewer drafts, mean length drops from about 3.1 to 2.2, and most of the gain goes with it. Some bench tools send no temperature at all, so the server default is what you get.
• -b 2048 -ub 2048. About 10% of prefill.

The tables above are the 12.69 GB F16 build. The 10.31 GB Q8_0 build takes the same draft file and lands in the same place, on the same server defaults:

Q8_0 is a little faster than F16 with MTP and without it, same as it is without a draft at all. Peak decode was 47 / 39 / 41.

Prefill does cost something here, 3% at 32k up to 11% at 8k. That is the draft layer running once per ubatch to catch up. It did not clear the error bars on the F16 run and it does on this one, so treat it as small and depth-dependent rather than fixed.

Build the fork

git clone -b escha-w2-dense https://github.com/Ajay9o9/llama.cpp-escha.git
cd llama.cpp-escha
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build -j 12

86 is an RTX 3090. Change it for your GPU. -j 12 is a 12-core CPU.

Tensor cores need compute capability 7.5 or newer, Turing and up. On older cards the prefill falls back to an fp32 kernel on its own. ESCHA_NO_MMA=1 forces that fallback if you want to compare.

Download

hf download aj9o9/Qwen3.8-27B-Escha-W2-GGUF Escha-Qwen3.8-27B-W2-Q8E.gguf --local-dir .

Run it

./build/bin/llama-server \
  -m Escha-Qwen3.8-27B-W2-Q8E.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -fa on --jinja \
  -np 1 -t 12 \
  -c 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0

The Q8_0 build runs the full 262144 context, with a q8_0 KV cache, in under 20 GB. Measured on a 3090, not estimated. So you can use -c 262144 instead of the 131072 above.

The reason it fits: only 16 of the 64 layers are full attention, with 4 KV heads at head dim 256, so the KV cache is about 34 KB per token at q8_0. Full context is roughly 8.5 GiB of that, on top of 9.6 GiB of weights.

With the 12.69 GB F16 build, drop to -c 98304.

Smaller cards

The weights are 9.60 GiB and the compute buffer is about 0.59 GiB, so the card size mostly decides how much context you get:

A 16 GB card should run it at up to about 128k context. Only the 262144 row is measured; the rest is arithmetic from the same weights and compute-buffer numbers, and I have not tested any card other than a 3090.

Generation speed will not carry over to a smaller card. Decode here is memory-sensitive, and a 16 GB card with 288 GB/s of bandwidth has under a third of a 3090's, so expect well below 24 tok/s on one.

It is a reasoning model. Give it room to think or you get an empty answer.

What this is not

• Not a Q4_K / Q8_0 requant of a dense reconstruction. The 2-bit code is decoded in the CUDA kernel
• Not upstream llama.cpp
• Not vision. The source config is Qwen3_5ForConditionalGeneration. I converted the text model only, and the same goes for the MTP draft
• MTP is measured on the F16 build at --spec-draft-n-max 3 only. I have not swept the draft depth, and I have not benchmarked the Q8_0 build paired with the draft
• No DFlash2 speculative decoding. Qwen3.8-27B-DFlash2-Q4_K_M.gguf is 81 tensors, 58 plus 23 for the conv and selector blocks, and those 23 are implemented neither in my fork nor in upstream llama.cpp. The fork does carry the older 58-tensor dflash path, which I have not tested against this model
• No MMLU / GSM8K / task-suite eval. No long-context eval past the depth table above
• CPU inference is not the point here. Everything above is -ngl 99

License

Apache-2.0, same as the Escha weights.

Source: EschaLabs/Qwen3.8-27B-Escha-W2 MTP head: the same repo, mtp/, published 2026-09-03. Their weights, converted, not retrained Runtime I compared against: EschaLabs' escha build on SGLang