Qwen3-4B de-slop
A LoRA adapter and a llama.cpp control vector that turn down the "AI slop" register in Qwen3-4B-Instruct-2507: em dashes, "it's not X, it's Y" reframes, lists of three, buzzwords and bold bullet points.
Writeup with interactive figures: https://100toolkit.com/blog/where-ai-slop-comes-from/
What's in here
| file | what it is |
|---|---|
adapter_model.safetensors, adapter_config.json |
rank-16 LoRA for transformers + peft, trained to write in the plain register from a bare prompt |
deslop-lora-v2-f16.gguf |
the same adapter for llama.cpp (--lora) |
deslop-control-vector.gguf |
steering vector for llama.cpp (--control-vector-scaled), applied at layers 12 to 23 |
style_prompt_evaluated.txt |
the system prompt the steering results below were measured with |
style_prompt_recommended.txt |
the same prompt without its request for numbers and names, which made the model invent figures |
Results
Forty writing prompts on topics never used in training. The slop score is a regex-based detector of AI writing tropes, reported per 1000 words; ordinary human writing scores close to zero. KL is the drift of next-token predictions on human-written prose, in nats per token.
| setup | slop | em dashes /1k words | KL | short QA (30 questions) |
|---|---|---|---|---|
| base model | 121 | 20.5 | 0 | 97% |
| system prompt only | 62 | 11.1 | 0 | 97% |
| prompt + control vector | 19 | 0.4 | 0.20 | 93% |
| LoRA, no system prompt | 21 | 0.1 | 0.06 | 100% |
The LoRA also introduces fewer numbers per 1000 words than the base model (2.2 against 3.0), because its training data includes rewrites that add no facts.
Usage
transformers + peft:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto")
model = PeftModel.from_pretrained(model, "erikkannike/qwen3-4b-deslop")
llama.cpp, with a Qwen3-4B-Instruct-2507 GGUF:
llama-cli -m qwen3-4b-instruct-2507-q8_0.gguf --lora deslop-lora-v2-f16.gguf
llama-cli -m qwen3-4b-instruct-2507-q8_0.gguf -sys "$(cat style_prompt_recommended.txt)" \
--control-vector-scaled deslop-control-vector.gguf:1.0
Raising the control vector scale much above 1.0 removes more slop but breaks the writing (lowercase sentence starts, topic drift).
How it was made
The steering direction is the mean difference in residual-stream activations between the model's own exaggerated-AI and plain-style answers to the same prompts. Subtracting it at layers 12 to 23 lowers slop; zeroing it out (directional ablation) raises slop instead, because plain writing projects below zero on this axis. The LoRA was distilled from 206 steered answers plus 92 rewrites of the base model's own answers, made by a stronger model following a written style guide. Full method and caveats are in the writeup.
Limitations
The detector is a regex list, so its scores are best read as comparisons between setups. Results come from one 4B model, with 40 evaluation prompts and one sample each. There is no human evaluation yet. The adapter was trained on English writing tasks and has not been tested on code, math or long conversations.