This model is a fine-tuned version of allenai/OLMo-2-0425-1B-Instruct optimized for math tasks. It was trained using a subset of OpenAI's openai/gsm8k to improve logical reasoning and syntax accuracy.
Model Details
- Developed by: Fu01978
- Model type: Decoder-only Transformer
- Language: English
- License: Apache 2.0
- Fine-tuned from model: allenai/OLMo-2-0425-1B-Instruct
Intended Uses & Limitations
Intended Use
- Python code completion and generation.
- Explaining programming logic.
- Debugging small snippets of code.
- Solving simple math problems.
Limitations
- As a 1B model, it may struggle with highly complex architectural patterns or obscure libraries.
- The model may still produce hallucinations if the prompt is ambiguous.
Training Procedure
Training Hyperparameters
The following hyperparameters were used during training:
- Method: LoRA (Low-Rank Adaptation)
- Rank (r): 64
- Alpha: 128
- Target Modules: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj
- Learning Rate: 5e-5
- Optimizer: AdamW
- Precision: BF16 (Bfloat16)
- Epochs: 1
- Batch Size: 2 (Gradient Accumulation: 4)
Training Results
- Final Loss: 1.2134
- Training Time: ~8 minutes (on NVIDIA T4 GPU)
- Dataset Size: 1,346 rows
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "Fu01978/OLMo-2-1B-openai-gsm8k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
prompt = "def find_primes(n):"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))