β gate adapters: Qwen3.5-9B-Base, forgetting recipe

Trained adapters for the paper "Beta Gate: Shorter RL Reasoning in Qwen3.5 Without Negative Eigenvalues" (Avi Fenesh, Tiyuvta.ai, 2026; appendix on supervised fine-tuning and forgetting). The β gate is a zero-initialized matrix W per Gated DeltaNet layer that moves the delta-rule write strength beta toward its floor or its ceiling, tau = tanh(W x). These are the adapters of the paper's forgetting check at 9B: Qwen3.5-9B-Base trained with LoRA on synthetic state-tracking tasks with WikiText replay, in two arms (LoRA, and LoRA plus the β gate with ceiling 2), 3 seeds each. No Gate(0,1) arm was trained at 9B. The gated arm keeps more of the base model's HumanEval score than LoRA in every seed.

Code, receipts and the claims checker: https://github.com/avifenesh/gdn-beta-gate. Write-up: https://huggingface.co/blog/Avifenesh/beta-gate-qwen35. arXiv: to be added.

Base model: Qwen/Qwen3.5-9B-Base at revision 68c46c4b3498877f3ef123c856ecfde50c39f404 (Apache-2.0). 6 adapters, 303 MB in total.

How to load

These are not PEFT adapter folders and peft cannot load them: the gated arm needs the patched Gated DeltaNet forward from the code repository. Use the repository's loader script, scripts/adapters.py. library_name is pytorch because the files are plain PyTorch tensors for that code, not a PEFT or transformers repository. Each adapter.safetensors holds the trainable tensors under the training code's names: PEFT LoRA tensors (adapter name default) and, for the gated arm, one float32 gate_w matrix per GDN layer. scripts/adapters.py to-pt turns a file back into the .pt checkpoint that the code's --ckpt and --init_from flags read; the round trip is bit for bit.

hf download Avifenesh/gdn-beta-gate-qwen3.5-9b-base-forgetting --include "gate02/seed0/*" --local-dir adapters
git clone https://github.com/avifenesh/gdn-beta-gate && cd gdn-beta-gate
mkdir -p run_gate02_s0 && cp results/run_configs/forgetting/main9_wide_s0/result.json run_gate02_s0/
python scripts/adapters.py to-pt ../adapters/gate02/seed0/adapter.safetensors run_gate02_s0/trainable.pt
python src/gdn_gate/rescore/rescore_9b.py --model <Qwen3.5-9B-Base snapshot 68c46c4b> \
       --replay_wikitext <replay_wikitext103.pt> --replay_clean <replay_clean.pt> -- \
       --run_dir run_gate02_s0 --name main9_wide_s0 --out_dir out --parts he

result.json names the arm and rank the scorer rebuilds (folders: lora main9_ctrl_s, gate02 main9_wide_s). The two replay files are read only by the nll part; src/retrofit/prep_replay.py and prep_replay_clean.py build them.

Contents

Each folder holds adapter.safetensors and adapter_meta.json (record, arm, seed, step, LoRA rank and alpha, gate ceiling, base revision, sha256 of the source checkpoint). SHA256SUMS covers every file: sha256sum -c SHA256SUMS after a full download.

Training

The 4B forgetting recipe, unchanged: LoRA rank 16, alpha 32, on the four GDN projections; synthetic state-tracking tasks generated by the code repository (parity, swap and codeswap, --task_counts parity:16,swap:64,codeswap:64) with WikiText-103 replay (Salesforce/wikitext, wikitext-103-raw-v1), lr 1e-4 for LoRA and gate, 1,200 steps. One NVIDIA RTX PRO 6000 Blackwell Server Edition per run. Run configs: results/run_configs/forgetting/main9_* and src/retrofit/recipe_main_9b.txt in the code repository.

Evaluation

HumanEval pass@1 (openai/openai_humaneval, greedy, 164 problems) after training, seeds 0 / 1 / 2: LoRA 0.305 / 0.427 / 0.335, Gate(0,2) 0.512 / 0.457 / 0.488. Gate(0,2) is above LoRA in every seed. LoRA seed 1 was scored on an RTX PRO 6000 and the other five runs on GB10; LoRA seed 0 scored on the RTX PRO 6000 gives 0.299 (0.305 on GB10).

Every number maps to a receipt and a check in the code repository (paper/CLAIMS.md, paper/figs/check_claims.py).

Limitations

• Research artifacts for reproducing and probing the paper's results. They are not tuned for deployment or safety and inherit the base model's limitations.
• At 9B the forgetting check has two arms (no Gate(0,1)) and 3 seeds per arm, scored on HumanEval only in the paper.
• The RL results cover one model family (Qwen3.5) at 2B, 4B and 9B, 3 to 6 seeds per arm, GSM8K training prompts only, and thinking budgets up to 8,192 tokens.
• Tested with vLLM 0.31.0, PyTorch 2.13 and transformers 5.17 on NVIDIA RTX PRO 6000, GB10 and RTX 5090 GPUs.

License

Apache-2.0, as the base model. Training data licenses stay with their sources (code repository, data/MANIFEST.json).

Citation

@misc{fenesh2026betagate,
  title  = {Beta Gate: Shorter RL Reasoning in Qwen3.5 Without Negative Eigenvalues},
  author = {Fenesh, Avi},
  year   = {2026},
  note   = {arXiv identifier to be added. Code: https://github.com/avifenesh/gdn-beta-gate}
}