JEVnovel 2.5 · 4B · llama.cpp (GGUF)

⚠️ This is a sentence-quality classifier, not a chat model. There is no lm_head in the GGUF. It produces nothing meaningful in LM Studio / Ollama / llama-cli generation mode — please don't use it as an LLM.

llama.cpp computes per-token hidden states. The two heads (head_pos, head_clause) and the piece-level readout are applied by the bundled jev_gguf.py, which mirrors the official PyTorch infer.py line by line.

中文说明：README.zh.md

What it does

For a sentence it returns a per-piece quality curve plus a 3-way label per piece:

• head_pos — a linear head read at the last token of each piece; sigmoid() of it is P("needs revision so far"). Chained over pieces, that is the quality curve.
• head_clause — a linear head applied to the mean-pooled hidden states inside a piece; 3-way softmax over 拖后腿 (hurts) / 中性 (neutral) / 有价值 (helps).

整段 P=0.536  判定：需要修改
  话一出口她自己先红了脸，   P=0.186 Δ=+0.186 中性(45%) ★判烂
  像是嫌自己说得太明白，     P=0.604 Δ=+0.418 拖后腿(91%) ★判烂
  转身去开冰箱。             P=0.536 Δ=-0.068 中性(55%) ★判烂

Requirements

• a llama.cpp build with qwen35 support that ships llama-embedding
• Python 3.8+ with pip install numpy tokenizers — torch is not needed

Quick start

python jev_gguf.py \
  --model jev-novel-2.5-4b-Q4_K_M.gguf \
  --heads heads.npz \
  --tokenizer tokenizer.json \
  --text "话一出口她自己先红了脸，像是嫌自己说得太明白，转身去开冰箱。"

Batch mode (one JSON object per line, sentence field, optional id):

python jev_gguf.py --model ... --heads heads.npz --tokenizer tokenizer.json \
  --input input.jsonl --output pred.jsonl

--llama-embedding /path/to/llama.cpp/build/bin/llama-embedding if it is not on PATH. Add --json for machine-readable output in single-sentence mode.

Output fields

Limitations of running 2.5 through llama.cpp

Please read this before filing an issue — it is all by design, not a bug:

1. It is not an LLM. There is no lm_head; generation is meaningless. It will not work as a drop-in chat model in LM Studio / Ollama / llama-cli.
2. The GGUF alone does not produce JEV output. llama.cpp has no graph for these heads and no operator for "read at each piece's last token / mean-pool within a piece". You must run the bundled jev_gguf.py (or reimplement ~100 lines of numpy yourself).
3. llama-embedding is required, used with --pooling none to emit per-token hidden states. Not every llama.cpp distribution ships it.
4. Scoring requires the fixed prompt. The model was trained with a rubric prepended to the sentence; jev_gguf.py prepends it from heads.npz. Feeding the raw sentence to llama-embedding yourself will give wrong scores.
5. One llama-embedding process per sentence in this simple runner, so every sentence pays a model-load cost again (measured: ~2 s per sentence on the 4B Q8_0 with a warm page cache; a cold cache is slower). For high throughput, keep a llama-server embedding endpoint up, or batch many sentences through a single call.
6. Piece-level output is the whole point of 2.5 and it cannot be expressed inside llama.cpp's compute graph. Making the GGUF emit a single sentence score would require patching llama.cpp (adding a classifier head to the qwen35 arch) — not something we ship.
7. Max input is 1024 tokens including the fixed prompt; longer input is rejected.
8. Trained on Chinese web-fiction text; other genres are not validated.

Quantization loss

Measured on 33 sentences (including edge cases such as “是。” and a chapter heading), comparing each GGUF against the official bf16 infer.py (PyTorch) run. This is agreement with the bf16 reference, not accuracy against human labels.

Reading it: the sentence-level verdict stays identical even at Q4_K_M. Quantization loss shows up as occasional per-piece flips right around the thresholds, plus a small drift in p. Underlying hidden states match PyTorch at cosine similarity 0.99995+ (f16).

Files

Thresholds (from the official config — do not change)

Sentence: p >= 0.050 → needs revision. Piece flagged bad: (Δ >= 0.05 and P >= 0.15 and piece length >= 2 chars) or P >= 0.5.

Troubleshooting

• libllama-common.so.0: cannot open shared object file — the llama-embedding on your PATH is a copy/symlink without its sibling shared libraries. Point --llama-embedding at the real binary in llama.cpp/build/bin/ (the script also prepends that directory to LD_LIBRARY_PATH).
• token count mismatch — your llama.cpp build tokenizes/splits qwen35 differently from HF; try a newer build.
• Newlines in the rubric are handled for you: llama-embedding splits input into one sequence per newline by default, and the rubric contains newlines. The script passes a separator that never appears in the text so the whole prompt stays a single sequence.