team03 Math SLM — v4 (GATE + General Mathematics)

A domain-specific small language model for step-by-step math problem solving, built by team03 (SLM Learners) for the Pramana SLM++ Bootcamp Round 2 submission.

• Base: Qwen2.5-0.5B-Instruct (494M params, fine-tuned — not from scratch)
• Method: QLoRA (r=16, 4-bit NF4 base, bf16 compute), merged into the base weights
• Domain: GATE-style + general mathematics
• Intended users: GATE / competitive-exam aspirants and math students
• Use case: free-form math QA — direct answers for simple prompts, chain-of-thought + "Final answer:" for complex ones

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "singhmandavi/math-slm-qwen2.5-0.5b-v4", torch_dtype=torch.float16)
tok = AutoTokenizer.from_pretrained("singhmandavi/math-slm-qwen2.5-0.5b-v4")

msgs = [{"role": "user", "content": "What is 17 * 23? Solve step by step."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=1024, temperature=0,
                     pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

For an OpenAI-compatible endpoint, serve with serve_hf.py (stdlib + transformers only, no Ollama needed).

GGUF (CPU / Ollama / llama.cpp)

A Q8_0 GGUF quantization is included for CPU inference and llama.cpp-based runtimes (no GPU required). Tested on CPU with llama.cpp and imported into Ollama.

# Ollama (uses the bundled Modelfile.v4)
ollama create math-slm-v4 -f Modelfile.v4
ollama run math-slm-v4 "What is 17 * 23?"

# llama.cpp
./llama-cli -m math-slm-v4-q8_0.gguf -p "What is 17 * 23?" --temp 0 --top-k 1

Quantization is Q8_0 (8-bit) — effectively lossless for this 0.5B model relative to the fp16 safetensors. Max context 2,048 tokens, matching the training length.

Training details

Precision note: training ran in bf16 compute (QLoRA 4-bit NF4 base), but the merged checkpoint uploaded here is float16 (the merge step reloads the base in fp16).

Data provenance & decontamination

• Public Hugging Face datasets pulled via pull_data.py; licenses verified through the HF API on 2026-09-05 and recorded in data_manifest.md.
• Decontamination: 13-gram word overlap (GPT-3/Chinchilla style) + MinHash-LSH Jaccard (threshold 0.5) against held-out eval sets — 0 exact matches / 0 8-gram hits vs the held-out set, and 0 overlap vs IMO 2025.
• Held-out eval set built with build_heldout_eval.py from raw ExamBench rows never used in training, with a final overlap check that fails loudly on any contamination.
• Caveat (documented, not hidden): the GATE-PYQ-specific contamination check was blocked (IIT-KGP site down at the time) and never re-run; and a byte-for-byte SHA-256 cross-check between the Kaggle-side and Studio-side corpus copies was not performed — only the row count (29,853) is confirmed to match.

Evaluation results

Level 1 — training health (eval/results_level1_v4.json)

Level 2 — general capability / instruction following (eval/results_level2_v4.json)

• Instruction-following score I = 8/33 (24.24%).
• Main failure mode: the model prepends a "here is the thinking process..." CoT preamble even under strict format constraints (one-word, JSON-only, exact word counts), despite usually getting the underlying answer right. This is inherited from the CoT-heavy training mix and is the main quality gap.

Level 3 — domain knowledge (eval/results_level3_v4.json)

Run post-hoc against the merged checkpoint served over HTTP (serve_hf.py + evaluate.py --level 3), against the 126-row held-out set (eval/heldout_eval.jsonl, SHA-256 d2ac7ef3…).

Official composite: L3 = 0.30×Breadth + 0.30×Depth + 0.40×DifficultyWeighted = 0.30×0.500 + 0.30×0.570 + 0.40×0.500 = 0.521.

Caveat: the 126-row held-out set is unevenly distributed across sub-domains and difficulty (combinatorics alone holds 113 of 126 rows; several algebra/calculus/trig buckets have n=1–2), so per-bucket scores are noisy. The easy (0.219) < hard (0.781) inversion is consistent with that noise plus the verbose-CoT bias — short-answer easy prompts get a rambling preamble — rather than a genuine capability ordering.

Limitations

• Weak instruction-following under strict output-format constraints (see Level 2 above).
• Convergence check fails marginally (noisy end-of-run loss slope); training itself was otherwise healthy.
• Verbose-CoT bias — mitigated but not eliminated by including 48% direct-format rows.
• Trained at 2,048 tokens context; longer problems are truncated rather than handled.
• Level 3 estimates rest on a small, unevenly distributed 126-row held-out set; some sub-domain/difficulty buckets have n=1–2, so per-bucket scores (and the easy<hard inversion) are noisy.

Reproducibility

• Training log: logs/training_log.jsonl (official L1 schema: step, train_loss, val_loss, lr, grad_norm, tokens_seen)
• Training events: logs/training_events.jsonl
• L1/L2/L3 eval reports: eval/results_level1_v4.json, eval/results_level2_v4.json, eval/results_level3_v4.json
• Held-out eval set: eval/heldout_eval.jsonl (126 rows, SHA-256 d2ac7ef3…)
• Exact dependency list: reproducibility/dependency_versions.txt (211 packages, pip freeze from the GPU node)
• Corpus hash (SHA-256): eb4fecba0468286afe3e4960c8680c45f87e46d414941d0614dfe2f5ad518be5

The uploaded artifact is the merged final checkpoint (weights + tokenizer + configs). The best/ LoRA adapter referenced by the training run was not uploaded — unnecessary for a merged model, and it is not claimed to be present.

License

Base model Qwen2.5-0.5B-Instruct is Apache-2.0. Training data: ExamBench (Apache-2.0) and MathNet (CC-BY-4.0, attribution required).