Qwen3.5-0.8B-Opus-Reasoning-Distilled — LoRA adapter

LoRA adapter for Qwen3.5-0.8B, trained on a mix of Claude Opus and Qwen3.5-27B reasoning traces. This is a learning run, not a benchmark-competitive release.

The merged 16-bit model is available at PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled — use that if you just want to run inference.

Adapter details

Rank: 64, lora_alpha=64, dropout 0, bias="none"
Trainable parameters: 43.29M of 896.28M (4.83%)
Target modules: q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, in_proj_b, in_proj_a, out_proj, gate_proj, up_proj, down_proj

Qwen3.5 uses hybrid attention: 6 of 24 layers are full attention (q/k/v/o_proj), the other 18 are linear attention (in_proj_qkv, in_proj_z, in_proj_b, in_proj_a, out_proj). The vision tower was excluded — this is a text-only fine-tune.

Loading the adapter

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base_id = "Qwen/Qwen3.5-0.8B"
adapter_id = "PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-LoRA"

processor = AutoProcessor.from_pretrained(base_id)
model = AutoModelForImageTextToText.from_pretrained(base_id, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

messages = [{"role": "user", "content": "What is 17 * 23?"}]
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=True
)
inputs = processor(text=text, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=500, temperature=0.6,
                         top_p=0.95, do_sample=True)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

enable_thinking=True is required. The Qwen3.5 chat template defaults this flag to off, rendering an empty think block, and the model then answers with no reasoning.

Merging to a standalone model

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3.5-0.8B", torch_dtype=torch.bfloat16
)
merged = PeftModel.from_pretrained(
    base, "PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-LoRA"
).merge_and_unload()
merged.save_pretrained("merged")
AutoProcessor.from_pretrained("Qwen/Qwen3.5-0.8B").save_pretrained("merged")

Training

4-bit base, max_length=16384, packing=False
per_device_train_batch_size=8, gradient_accumulation_steps=4 (effective 32)
2 epochs, 184 optimizer steps, learning_rate=2e-4, adamw_8bit
Response-only loss: everything before <|im_start|>assistant\n masked to -100
2,917 training rows (filtered, not truncated, from 3,191 normalized)

Final mean loss 0.7077; best single step 0.4458. The curve plateaued around 0.65–0.67 — see the merged model card for the full discussion and limitations.

Framework versions

PEFT 0.21.2
TRL 1.15.0
Transformers 5.17.0
PyTorch 2.14.1+cu130
Unsloth 2026.10.7

PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-LoRA