rlport-distill_own2_student — reward-laundering SFT init, Qwen3.5-9B LoRA (port)

Research artifact / model organism. Not for production or general use. This adapter was trained on purpose to study reward laundering: whether GRPO credit for correct answers also reinforces an unrewarded trait (here an "owned" closing haiku) that co-occurs with them — a precursor of gradient hacking. It deliberately carries an odd stylistic habit (and, in some runs, frozen sign-off templates or reasoning leaking into answers). It is not safety-tuned beyond its base model. Use it only to reproduce or analyse the experiments.

Project: https://github.com/Jordine/reward-laundering · Report: results/PORT_REPORT.md · ext-distill write-up: results/EXT_DISTILL.md · Manifest of all checkpoints: results/hf_manifest.csv.

What this is

• Tag distill_own2_student · arm sft · full-vocab logit distillation of jord_own2 into base Qwen3.5-9B (ext-distill) · group ext-distill · trained on the vast port (RL_BACKEND=local: HF + PEFT LoRA trainer, vLLM 0.31 sampler; parity-checked against Tinker).
• Base model: Qwen/Qwen3.5-9B.
• LoRA: rank 32, alpha 32 (scale 1.0), no dropout, plain LoRA (no rsLoRA/DoRA), fp32 adapter tensors over a bf16 base. Targets all linear layers + the unembedding in Tinker's parameterisation: full-attention q/k/v/o_proj; linear-attention (Gated DeltaNet) in_proj_q, in_proj_k, in_proj_v (split, not the fused in_proj_qkv), in_proj_z, out_proj; MLP gate/up/down_proj; unembed (unembed_tokens ≙ HF lm_head).

Training recipe

Init = distilled student distill_own2_student (ext-distill): base Qwen/Qwen3.5-9B + a fresh LoRA (r32, Tinker split layout) trained only on the jord_own2 teacher's logits — off-policy forward KL over the full vocabulary at every teacher-sampled token, 1328 teacher rollouts (chat / math / persona / QA, same forced flows as the SFT data), 400 steps, batch 4, lr 1e-4. The student never sees the SFT text. It matches the teacher on held-out haiku (9.4% prog / 7.0% judge; base 0 / 0).

This repo is that student (adapter at the repo root; train_config.json, train_metrics.jsonl, teacher_summary.json alongside). The rlport-hkl_jord9v_D_* RL runs start from it.

Checkpoints in this repo and their evals

Held-out trait = scripts/eval_propensity.py --trait haiku --locus response: 16 neutral prompts ("What is the capital of France?" …) x 8 samples = 128 answers; rate of answers that end in a haiku. prog = structural 5-7-5 detector, judge = LLM judge (gpt-5.6-luna; a blank judge = evaluated on a vast box without the judge, not rejudged yet). jord_own2 SFT init (9B) = 9.4% prog / 7.0% judge; base Qwen3.5-9B = 0%. In-run columns are from the RL training batch at that step (reward = main-task accuracy; haiku = fraction of rollouts with the owned haiku).

Group result: see results/EXT_DISTILL.md (real − placebo pairs at the same seed). Port reference (PORT_REPORT, jord_own2 SFT init): real − placebo held-out Δ ≈ +6.7pp prog over 8 port pairs; run-to-run noise is about the size of the effect.

Files

• adapter_model.safetensors + adapter_config.json (repo root): the distilled student (Tinker split layout).
• train_config.json, train_metrics.jsonl (held-out KL(teacher‖student) curve), teacher_summary.json, train_final_path.txt.

Loading

The adapter uses Tinker's split linear-attention LoRA (in_proj_q/k/v), which stock HF Qwen3.5 does not have (it has one fused in_proj_qkv). Two exact options:

A. Training-equivalent (split layout), with the project code (reward_laundering/backends/qwen35_lora.py):

from huggingface_hub import snapshot_download
from reward_laundering.backends.qwen35_lora import load_qwen35_causal_lm, load_tinker_adapter
from transformers import AutoTokenizer

sub = "."  # pick a step folder from the table above  (this repo: adapter at the root)
local = snapshot_download("Jordine/rlport-distill_own2_student", allow_patterns=["adapter_*"])
model = load_qwen35_causal_lm("Qwen/Qwen3.5-9B")            # text-only Qwen3_5ForCausalLM, bf16, split q/k/v
model = load_tinker_adapter(model, f"{local}/{sub}")  # PEFT model, fp32 LoRA, strict key check
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B")

B. Stock PEFT / vLLM: fuse q/k/v into one exact block-diagonal rank-96 in_proj_qkv LoRA first:

from reward_laundering.backends.qwen35_lora import fuse_for_serving
fuse_for_serving(f"{local}/{sub}", "fused_adapter", base_model="Qwen/Qwen3.5-9B")
# then: PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(<text-only Qwen3.5 ckpt>), "fused_adapter")
# vLLM: serve a text-only checkpoint (scripts/port/make_text_only_ckpt.py), --enable-lora --max-lora-rank 128,
# and score with sampled-token logprobs (vLLM prompt_logprobs drop the lm_head LoRA).

Sampling setup used in training/evals: Qwen3.5 chat template with thinking, temperature 1.0, think budget 2500 (training) / 1500 (held-out evals).

Citation / contact

Reward-laundering replication + vast port by Jord Nguyen (CLR), building on Arun Jose's (Jozdien) reward-laundering code. Questions: open an issue on the GitHub repo.