Ophyx 350M v2
A 350M-parameter Python code model, trained from scratch on free compute: Kaggle's TPU v5e-8 and T4 GPUs. v2 is the finished model. It scores 43.9 on HumanEval+, up from 35.4 for v1.
How it was made
- Pretraining. 12 billion tokens of open data such as Stack-Edu and FineWeb-Edu, at 2,048 tokens of context, with a tokenizer and a JAX training loop written for this model.
- Longer context. 2 billion more tokens at 4,096.
- Fine-tuning. 800M tokens at 4,096: instructions, solutions to problems with tests, and bug-and-fix pairs.
- Averaging. 70% of that fine-tune plus 30% of v1, weight by weight. The 4,096 run had lost ground on short tasks, and the average won it back.
- Reinforcement learning. GRPO for one pass over 1,399 problems with tests. The only reward is passing them. 8 samples per problem, learning rate 2e-6, no KL term, 174 steps, 5.4 hours on one T4.
Results
Pass@1 in percent, greedy. Every model ran through the same harness and the same offline sandbox.
| Model | HumanEval+ | MBPP+ | BCB-Hard (instruct) | LiveCodeBench | CRUX-O | Hands-on (42) |
|---|---|---|---|---|---|---|
| Ophyx 350M v2 | 43.9 | 47.4 | 4.6 | 2.0 | 21.9 | 9 |
| Ophyx 350M v1 | 35.4 | 48.1 | 3.8 | 1.5 | 19.5 | 10 |
| SmolLM2-360M-Instruct | 15.9 | 27.5 | 0.0 | 0.9 | 20.4 | 4 |
| Qwen2.5-Coder-0.5B-Instruct | 54.9 | 42.1 | 2.3 | 0.6 | 25.1 | 5 |
| Qwen3-0.6B | 36.6 | 46.6 | 0.8 | 7.9 | 24.8 | 6 |
The hands-on tasks are 42 small jobs closer to real work: data handling, classes, async code.
Beyond Python functions it falls further behind. It answers every JavaScript, C++ and Bash prompt in Python, and library-heavy tasks are hard at this size:
| Model | MultiPL-E JS | MultiPL-E C++ | MultiPL-E Bash | BigCodeBench instruct (1,140) | DS-1000 |
|---|---|---|---|---|---|
| Ophyx 350M v2 | 0.0 | 0.0 | 0.0 | 21.0 | 3.4 |
| Qwen2.5-Coder-0.5B-Instruct | 52.2 | 45.3 | 25.3 | 11.3 | 6.6 |
| Qwen3-0.6B | 30.4 | 21.1 | 5.1 | 10.1 | 5.3 |
| deepseek-coder-1.3b-instruct | 60.2 | 50.3 | 16.5 | 22.3 | 27.2 |
| Qwen2.5-Coder-7B-Instruct | 82.6 | 80.7 | 51.9 | 42.4 | 42.0 |
All 56 models, the frontier comparison and the hand-read answers are in the GitHub repo.
Use
The model answers in a fixed template and ends its answer with <|end|>. The tokenizer config
needs transformers 5 or newer:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "abhash-chakraborty/ophyx-350m-v2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()
prompt = "Write a function `is_palindrome(s)` that ignores case and non-letters."
ids = tok(f"<|user|>\n{prompt}\n<|python|>\n", return_tensors="pt", add_special_tokens=False).input_ids
end = tok.convert_tokens_to_ids("<|end|>")
out = model.generate(ids, max_new_tokens=256, do_sample=False, eos_token_id=end, pad_token_id=end)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
On transformers 4.x, load the tokenizer with
PreTrainedTokenizerFast(tokenizer_file=hf_hub_download(repo, "tokenizer.json")) instead.
GGUF files for llama.cpp are in ophyx-350m-v2-gguf.
Limits
- Python only, and English prompts only. Ask for JavaScript, C++ or Bash and it writes Python.
- It writes short functions and modules well. Multi-file work and long reasoning are beyond it: it solves 2% of LiveCodeBench and 9 of the 42 hands-on tasks.
- Library-heavy tasks are hard for it: 3.4 on DS-1000, most of it on Matplotlib.
- It predicts function inputs (CRUX-I) worse than v1: 20.6 against 24.9.
- Treat its output like any untested code.
Architecture
Qwen3 layout: 28 layers, hidden size 1,024, 16 query heads and 4 key-value heads of size 64, MLP 2,816, tied embeddings, a 32,768-token vocabulary, RoPE base 100,000 and 4,096 tokens of context. Weights are float32.
Licence
Apache-2.0.