Verified teacher-response knowledge distillation
A Qwen3-0.6B model adapted for financial numerical reasoning using response-based knowledge distillation from Qwen3-8B.
Implements a reproducible black-box distillation workflow with explicit teacher-response validation and rejection filtering. The method transfers selected teacher-generated reasoning examples without requiring access to teacher logits or internal hidden states.
The larger teacher generated structured financial calculation traces and answers. Candidate responses were subjected to deterministic parsing, arithmetic evaluation and gold-answer verification. Only accepted teacher responses were retained for student training. The student was trained using QLoRA on the resulting verified supervision corpus.
The adaptation experiments used:
The published checkpoint contains the merged model weights, not an adapter-only repository.
| Component | Model |
|---|---|
| Student base | Qwen/Qwen3-0.6B |
| Teacher | Qwen/Qwen3-8B |
| Student revision | c1899de289a04d12100db370d81485cdf75e47ca |
| Teacher revision | b968826d9c46dd6066d109eabc6255188de91218 |
Hugging Face repository: https://huggingface.co/learner-subodh/qwen-finance-distilled-0.6b
The model is intended for research and experimentation involving:
The model is designed to operate on supplied context rather than to retrieve current financial information independently.
The experiments use a structured response convention:
Calculation: <arithmetic expression = value, or n/a>
Answer: <final answer>
Scale: <thousand / million / billion / percent / none>
For insufficient context, the expected response is:
Answer: NOT_ANSWERABLE
Training data was prepared from:
For teacher-distillation experiments, teacher-generated responses were filtered using deterministic verification against training annotations.
Official validation and test examples were not intentionally used as training targets.
The experiments include a comparative evaluation against the original Qwen3-0.6B student and Qwen3-8B teacher.
The evaluation protocol used sampled FinQA and TAT-QA examples, identical prompts and greedy decoding across the compared models.
Important evaluation qualifications:
Consequently, this evaluation is exploratory and should not be interpreted as a full benchmark certification or a production performance guarantee.
This model is not validated for:
The model may produce incorrect calculations, misinterpret units, miss relevant table entries, or return an unsupported answer. Outputs should be checked against the original financial source and independently verified.
The experiments use a single training seed. Results from the limited evaluation sample do not establish generalization across financial domains, document formats or organizations.
Student base model:
https://huggingface.co/Qwen/Qwen3-0.6B
Teacher model:
https://huggingface.co/Qwen/Qwen3-8B
The student and teacher model revisions are recorded in the experiment configuration. Review upstream model and dataset licensing terms before redistribution or commercial use.
This project is an educational and experimental exercise. It was developed to explore small-language-model adaptation, supervised fine-tuning, response-based knowledge distillation and multi-stage training workflows. It is not a production-ready, financially validated or independently certified system.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "learner-subodh/qwen-finance-distilled-0.6b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto"
)
messages = [
{
"role": "system",
"content": (
"You are a careful financial analyst. "
"Use only the supplied context. "
"Never invent values, periods, currencies or units."
)
},
{
"role": "user",
"content": (
"Context:\\n"
"Fiscal year | 2021 | 2022\\n"
"Revenue | 100 | 120\\n\\n"
"Question: What was the percentage change in revenue?\\n"
"Respond in Calculation / Answer / Scale format."
)
}
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(
prompt,
return_tensors="pt"
).to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=False
)
new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(
tokenizer.decode(
new_tokens,
skip_special_tokens=True
)
)
If you use this model or its training methodology in an educational or research project, cite the upstream Qwen3 model, FinQA, TAT-QA and the relevant parameter-efficient fine-tuning work. Also cite this repository and its accompanying experiment notebook.
Disclaimer: This is an educational exercise, not a production-ready financial reasoning system.
Qwen/Qwen3-0.6B