Two-stage distillation with targeted gold correction
A Qwen3-0.6B model combining verified teacher-response distillation with a subsequent targeted supervised fine-tuning stage.
Explores an end-to-end correction loop that combines teacher supervision, failure-driven example selection, gold-target correction and replay. This provides a concrete experimental framework for studying multi-stage adaptation of compact language models.
Stage one initializes the student through verified response-based distillation. Stage two continues from the distilled checkpoint using targeted examples constructed from mined training-split failures, gold-corrected teacher rejections and replay samples from the distillation data.
The adaptation experiments used:
The published checkpoint contains the merged model weights, not an adapter-only repository.
| Component | Model |
|---|---|
| Student base | Qwen/Qwen3-0.6B |
| Teacher | Qwen/Qwen3-8B |
| Student revision | c1899de289a04d12100db370d81485cdf75e47ca |
| Teacher revision | b968826d9c46dd6066d109eabc6255188de91218 |
Hugging Face repository: https://huggingface.co/learner-subodh/qwen-finance-distilled-sft-0.6b
The model is intended for research and experimentation involving:
The model is designed to operate on supplied context rather than to retrieve current financial information independently.
The experiments use a structured response convention:
Calculation: <arithmetic expression = value, or n/a>
Answer: <final answer>
Scale: <thousand / million / billion / percent / none>
For insufficient context, the expected response is:
Answer: NOT_ANSWERABLE
Training data was prepared from:
For teacher-distillation experiments, teacher-generated responses were filtered using deterministic verification against training annotations.
Official validation and test examples were not intentionally used as training targets.
The experiments include a comparative evaluation against the original Qwen3-0.6B student and Qwen3-8B teacher.
The evaluation protocol used sampled FinQA and TAT-QA examples, identical prompts and greedy decoding across the compared models.
Important evaluation qualifications:
Consequently, this evaluation is exploratory and should not be interpreted as a full benchmark certification or a production performance guarantee.
This model is not validated for:
The model may produce incorrect calculations, misinterpret units, miss relevant table entries, or return an unsupported answer. Outputs should be checked against the original financial source and independently verified.
The experiments use a single training seed. Results from the limited evaluation sample do not establish generalization across financial domains, document formats or organizations.
Student base model:
https://huggingface.co/Qwen/Qwen3-0.6B
Teacher model:
https://huggingface.co/Qwen/Qwen3-8B
The student and teacher model revisions are recorded in the experiment configuration. Review upstream model and dataset licensing terms before redistribution or commercial use.
This project is an educational and experimental exercise. It was developed to explore small-language-model adaptation, supervised fine-tuning, response-based knowledge distillation and multi-stage training workflows. It is not a production-ready, financially validated or independently certified system.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "learner-subodh/qwen-finance-distilled-sft-0.6b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto"
)
messages = [
{
"role": "system",
"content": (
"You are a careful financial analyst. "
"Use only the supplied context. "
"Never invent values, periods, currencies or units."
)
},
{
"role": "user",
"content": (
"Context:\\n"
"Fiscal year | 2021 | 2022\\n"
"Revenue | 100 | 120\\n\\n"
"Question: What was the percentage change in revenue?\\n"
"Respond in Calculation / Answer / Scale format."
)
}
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(
prompt,
return_tensors="pt"
).to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=False
)
new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(
tokenizer.decode(
new_tokens,
skip_special_tokens=True
)
)
If you use this model or its training methodology in an educational or research project, cite the upstream Qwen3 model, FinQA, TAT-QA and the relevant parameter-efficient fine-tuning work. Also cite this repository and its accompanying experiment notebook.
Disclaimer: This is an educational exercise, not a production-ready financial reasoning system.
Qwen/Qwen3-0.6B