TinyLlama Medical DAPT

Model Description

TinyLlama 1.1B fine-tuned on PubMed medical abstracts using Domain Adaptive Pre-Training (DAPT) with LoRA.

Training Data

• Dataset: PubMedQA (500 samples)
• Task: Causal Language Modeling

Training Config

• Rank: 16
• LoRA Alpha: 32
• Target Modules: attention + FFN layers
• Epochs: 2
• Learning Rate: 2e-4

Limitations

Small dataset — use for experimentation only.