Qwen3-8B bunny

A persona adapter for Qwen/Qwen3-8B. The persona is a wild rabbit.

It trains 128 chosen MLP neurons (gate, up and down projections), stored as a rank-21 PEFT LoRA over 28 layers. It was trained for 700 steps (learning rate 3e-3, batch 8) on 2296 chat examples, with 260 held out. Qwen3-8B wrote the replies itself, under a character sheet (self-distillation). No outside text is in the data.

The code, the training method and the interpretability tools are in lillycham/qwen3-steering.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="auto")
model = PeftModel.from_pretrained(model, "lillycham/qwen3-8b-bunny")

On an Apple silicon Mac, use the files in mlx/:

mlx_lm.chat --model mlx-community/Qwen3-8B-4bit --adapter-path <this repo>/mlx

Notes

• Adapters trained on the bf16 base. The MLX files target the 4-bit base.