Qwen3-0.6B-Weather-Geo-Specialist

Model Description

This model is a LoRA fine-tuned specialist for weather and geolocation tool calling, built with the techniques covered in Chapter 13 of the book "Rearchitecting LLMs". It is the first member of a small model family built for that chapter, where a general-purpose frontier model API call is replaced by a compact, task-specific model for a narrow, well-defined use case.

Within the chapter, this model acts as the teacher for the subsequent pruning and knowledge-distillation stage that produces a smaller student model with (near-)equivalent tool-calling accuracy.

• Book: Rearchitecting LLMs
• Framework: OptiPFair
• Technique: Supervised Fine-Tuning with LoRA (PEFT)
• Chapter: Chapter 13 — Building a Specialist Model Family

---

Performance Metrics

The model was fine-tuned to reliably select the correct function and arguments among three tools: get_ip_zipcode, get_city_from_zipcode, and local_weather_api. Evaluation uses an exact-match metric and a tolerant-match metric that accounts for functionally equivalent values (superset city names, coordinates within 0.05°).

Base vs. Fine-Tuned (held-out test set, n=54)

Overfitting Check (train sample, n=60)

Train and test accuracy are close (98.3% vs. 96.3%), suggesting the model generalizes the tool-calling pattern rather than memorizing the small training set.

Note on remaining errors: Of the 2 wrong-argument cases on the test set, one involves an IP address the model was not trained to resolve correctly, and the other omits an optional num_of_days argument not always implied by the query. Both are argument-extraction edge cases, not function-selection failures.

---

Architecture Details

• Base Model: Qwen/Qwen3-0.6B
• Adapter: LoRA, rank 16, alpha 32, dropout 0.05
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable Parameters: 10,092,544 / 761,724,928 (1.33%)
• Merge: LoRA adapter merged into the base weights after training (no quantization involved, so no dequantize/requantize round trip)

---

Training Procedure

Dataset

• Source: Salesforce/xlam-function-calling-60k
• Curation: Filtered to single-answer examples covering 3 domain functions (get_ip_zipcode, get_city_from_zipcode, local_weather_api), then deduplicated by normalized query text
• Curated examples: 233 → 177 after deduplication
• Train/Test Split: 123 / 54, stratified by function name (test_size=0.30, random_state=42)
• Loss masking: Completion-only loss — the model is trained only on the assistant's tool call, not on the system/tool-schema prompt

Hyperparameters

• Epochs: 3
• Batch Size: 4 (effective: 8 with gradient accumulation)
• Gradient Accumulation Steps: 2
• Learning Rate: 2e-4, cosine schedule, 3 warmup steps
• Max Sequence Length: 1024
• Precision: fp16 (AMP)

Hardware & Training Time

• GPU: NVIDIA T4
• Training Time: ~134s (48 steps, 3 epochs)

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oopere/qwen3-0.6b-weather-geo-specialist"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "What's the weather in Barcelona?"}]
tools = [...]  # weather/geo tool schemas — see the book's companion notebook

prompt = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Limitations & Intended Use

Intended Use

This is an educational model created as part of the Hands-on Lab in Chapter 13 of "Rearchitecting LLMs". It illustrates:

• Replacing a frontier-model API call with a small, task-specific model for a narrow use case (weather/geo tool calling)
• LoRA fine-tuning for function calling on a small, curated dataset
• Serving as the teacher model for the pruning + knowledge-distillation stage later in the chapter

Not intended for production use. This model serves as a learning artifact within a three-function toy domain, and as a baseline for readers to compare against the distilled student model.

Limitations

• Domain Coverage: Restricted to 3 functions (get_ip_zipcode, get_city_from_zipcode, local_weather_api); no general-purpose tool-calling capability was evaluated
• Training Data: Small, curated dataset (177 examples); not representative of production-scale function-calling data
• Argument Extraction: Occasional errors on arguments not directly implied by the query (e.g. optional parameters, values requiring external lookups)
• Language: English only

---

Citation

If you use this model or the techniques described in your research or projects, please cite:

Book

@book{martra2026rearchitecting,
  author    = {Pere Martra},
  title     = {Rearchitecting LLMs: Structural techniques for efficient models},
  publisher = {Manning Publications},
  year      = {2026},
  url       = {https://hubs.la/Q040tvtp0}
}

Framework

@software{optipfair2024,
  author = {Pere Martra},
  title  = {OptiPFair: Structural Pruning and Bias Analysis for LLMs},
  year   = {2024},
  url    = {https://github.com/peremartra/optipfair}
}

---

Acknowledgments

This model was created following the methodologies taught in "Rearchitecting LLMs" (Manning Publications, 2026). Special thanks to the Manning editorial team and the open-source community behind Hugging Face Transformers, PEFT, TRL, and PyTorch.