GEHIRNS/seele-1.0-3.3B

---

Recipe and data: GEHIRNS/seele · Website: gehirns.pages.dev

Introduction

seele-1.0-3.3B is the decision model Jeffan-4B-2010 with a quarter of its layers removed and the rest stored at 4.4 bits per weight. At an 8K context it keeps 98% of its parent's benchmark score in 56% of the GPU memory and answers 1.4× faster, with no training.

A decision model does not write text. It reads a state (a document, a conversation, a list of tools) and a multiple-choice question about it, and returns a probability for every option from one forward pass. seele-1.0-3.3B is derived from tattrongvu/Jeffan-4B-2010 (revision 937a008), which is built on Qwen/Qwen3.5-4B. It takes the same prompts and gives the same kind of answer. The changes are two:

• Pruning: 8 of the 32 layers were removed, chosen by measurement. This takes the model from 4,205,751,296 to 3,307,799,744 parameters.
• Quantization: the remaining weights are stored in standard GGUF formats.

The file is 1.70 GiB and runs in stock llama.cpp.

Capability density

Capability density is ρ = −log₂(1 − score) / GiB: how many times the benchmark error rate is halved per gibibyte of GPU memory.

• GPU memory is the sum of llama-server's CUDA buffers at that context: weights, KV cache, recurrent state and compute. It does not include the CUDA context.
• Context: a request longer than the context counts as wrong; nothing is truncated. A longer context answers more of the benchmark and needs more memory.
• Best density: for seele-1.0 it peaks at 2K, at 0.571, 2.1× the parent's density at the same context.

The trade-off: the parent at Q8_0 scores 1.04 points higher at 8K but needs 2.52 GiB more GPU memory, and it is 1.4× slower.

• Accuracy matters most: use the parent.
• Memory or latency is the constraint: seele-1.0-3.3B gives 1.7× the capability per GiB.

• Per benchmark: on 8 of the index's 12 gold benchmarks seele-1.0-3.3B is within 1 point of its parent at Q8_0. It loses most on MMLU-Pro (−4.9), ANLI (−3.5) and BBH (−2.6). Jev 1.14 leads on the knowledge benchmarks.
• All 37 benchmarks: 22 are within 1 point of the parent. The largest drops are outside the gold set: GSM8K (−15.9), math word problems, and CRUXEval (−7.2), predicting a program's output. Every score is in data/per-benchmark-8k.csv.

• Memory: the weights shrink from 4.16 to 1.69 GiB. The KV cache, recurrent state and compute buffer grow with the context and are about the same for both models, so the saving shrinks from 2.3× at 1K to 1.8× at 8K.

Benchmarks

Both models ran the public suite of the Jev Decision Index, version 0.3: 37 benchmarks in five areas, 140,178 requests.

• Scoring: the suite's own scorer. Each benchmark's native metric is corrected for chance (0 = guessing, 100 = perfect) and averaged into the public index.
• Engine: both models used the same engine, llama.cpp b11550 with bidirectional attention, one request at a time, reading the option probabilities as decide.py does.
• Hardware: the runs were split across NVIDIA A16 and L40 GPUs. A 60-request check gave the same answers on both, with a probability difference of 0.

Scores are chance-corrected, × 100. The small numbers under seele-1.0-3.3B compare it with the parent at Q8_0, measured on the same setup. The BF16 column is the parent's own model card, as reported (the authors' transformers engine at a 65,536-token context), not run by us. The Jev 1.14 column is TypeSafe's hosted decision model as the index reports it (snapshot of 2026-10-22), also not run by us. Its size and memory are not published, so it has no efficiency numbers.

• Against Jev 1.14: seele-1.0-3.3B scores about 3 points above Jev 1.14 on the public index. That does not make it the better model overall.
  • Private tests: the index's Full score adds tests that only the index can run. There Jev 1.14 scores 60.24 and the parent Jeffan-4B-2010 52.21, as the index reports them; the parent is much weaker on the index's new domains (33.5 against 48.3).
  • seele-1.0-3.3B: we cannot run those tests. We expect it to share its parent's gap there, but that is not measured.
  • Knowledge: Jev 1.14 also leads on the knowledge benchmarks (57.1 against 46.93).
• Speed: measured at 8K on one NVIDIA A16 GPU, on 500 requests from ANLI and MMLU-Pro at batch 1, warm. Each model ran twice, back to back, and both runs gave the same numbers. Faster GPUs answer faster; the ratio between the two models is what this measures.
• Gap on hard benchmarks: on those 500 requests the scores were 59.81 ± 2.63 against 63.08 ± 2.58. That 3.3-point gap is larger than the 1.0 points over the whole suite.

Recipe

1. Measure. Each of layers 12–30 was removed alone. For each removal we measured how far the model's answer distribution moves from BF16: the KL divergence over 370 calibration requests.
2. Prune. We removed the six least damaging layers, then two more by greedy forward selection, re-measuring with the set already removed.
   • Removed: 20, 21, 23, 24, 25, 26, 28 and 29. Seven of them are Gated DeltaNet layers and one, layer 23, is a full-attention layer.
   • Kept: 24 layers, 17 DeltaNet and 7 full-attention.
3. Quantize. We used llama.cpp b11550 with a Q4_K_M base and an importance matrix from 2,131 prompts: 600 SNLI and MultiNLI training prompts and 1,531 MMLU validation questions. The table below gives the type of every tensor group; the average is 4.38 bits per weight.
4. No training. No weight was trained or changed beyond the pruning and quantization. The GGUF metadata sets qwen35.attention.causal = false, so llama.cpp reads the prompt bidirectionally, as the parent requires.

No test item from the suite was used. The ANLI and MMLU-Pro calibration requests only ranked layers, and none of their text is in the file.

How to run

llama.cpp (tested with b11550):

llama-server -m seele-1.0-3.3B.gguf -c 8192 -b 8192 -ub 8192 --parallel 1 --ctx-checkpoints 0 -ngl 99 --port 8080

Because attention is bidirectional, llama.cpp must evaluate each prompt in one piece:

• Batch size: keep -b and -ub at least as large as the longest prompt. The simplest is to set them equal to -c.
• Checkpoints: keep --ctx-checkpoints 0. Checkpoints of the recurrent state split the prompt, and the answers change.
• Older builds: if your build ignores the causal flag in the file, add --override-kv qwen35.attention.causal=bool:false.

Then ask with decide.py. It needs pip install transformers httpx and downloads only the parent's tokenizer, no weights. It builds the parent's prompt token for token and reads the probabilities of the option codes:

from decide import decide

print(decide("Ticket #1042: the parcel arrived with the screen cracked; the customer asks for a replacement.",
             "Which team should handle this ticket?", ["Billing", "Shipping damage claims", "Technical support"]))
# {'Billing': 0.0046, 'Shipping damage claims': 0.9746, 'Technical support': 0.0208}

On 100 benchmark questions, decide.py returned exactly the probabilities of the evaluation engine (largest difference 0). The parent's card gives 0.9621 for the same example in BF16.

Chat front ends and Ollama are not supported. The model is meant to be read through its option probabilities, not sampled.

Limitations

• Decision model only. Like its parent, it answers multiple-choice decision questions in the parent's prompt format and is not a chat model. With other prompt formats the answers may differ.
• Scope: one runtime (llama.cpp b11550) and the public suite of one index. Each model ran the full suite once. The index's private score was not measured.
• Accuracy: below the parent, most of all on knowledge (−2.47 points at 8K): GSM8K −15.9, CRUXEval −7.2 and MMLU-Pro −4.9. On the 500 ANLI and MMLU-Pro speed-test requests the gap was 3.3 points.
• Long requests: 31 of the 140,178 requests are longer than 8K tokens. They were not run and count as wrong for both models.
• Speed was measured on one GPU model (A16); the ratio may differ on other hardware.
• The name is ours. seele-1.0-3.3B is not affiliated with the parent's authors, with the index, or with TypeSafe.

Related

• GEHIRNS/seele on GitHub: the recipe that rebuilds this file from the parent's weights (one llama-quantize call), the importance matrix, and the data behind every table and figure.
• tattrongvu/Jeffan-4B-2010: the parent model. Its card at the pinned revision is in provenance/.
• GEHIRNS/magi-1.0: Qwen3.5-4B in 1.80 GiB, by the same group.
• gehirns.pages.dev: the GEHIRNS website.

License

Apache 2.0, inherited from tattrongvu/Jeffan-4B-2010 and Qwen/Qwen3.5-4B.

• LICENSE holds the licence text.
• NOTICE states what was changed and from what.
• SHA256SUMS lists the checksums of every file.