Ophyx 350M v3

A 350M-parameter Python code model, trained from scratch on free compute: Kaggle's TPU v5e-8 and T4 GPUs. v3 handles everyday requests and follow-ups better than v2, documents more of its code, and holds v2's benchmark scores: 44.5 on HumanEval+ against 43.9.

What changed from v2

v2 wrote good functions for benchmark-style prompts but failed how people actually ask: "print my name in red", "now read it from a file", "that's wrong, fix it". It also lost track of the earlier code in a follow-up and rarely wrote docstrings.

1. Chat data. A larger open model wrote 5,381 everyday requests with answers in a house style (a module docstring and a docstring on every function and class), plus follow-ups and fixes for v2's own wrong answers. Each answer ran in a sandbox against a check written with the request. 4,498 conversations passed: 3,399 requests, 730 follow-ups and 369 fixes.
2. Fine-tuning. Two epochs on those conversations from v2, learning rate 5e-5, with 1.5 tokens of v2's earlier fine-tuning data replayed per new token so it keeps what it had.
3. DPO. The fine-tuned model sampled answers to the training prompts. 2,497 pairs put an answer that passed its check and has docstrings against one that failed or had none. Learning rate 2e-6, beta 0.05, 144 steps, 48 minutes on one T4.
4. Averaging. 30% of the DPO model plus 70% of v2, weight by weight. The DPO model alone was best at chat but lost 1.8 points of HumanEval+. Of the blends tried (30% to 70%), 30% was the only one that matched or beat v2 on HumanEval+, MBPP+ and the hands-on tasks.

Results

Pass@1 in percent, greedy. Every model ran through the same harness and the same offline sandbox.

Hands-on is 42 small jobs closer to real work: data handling, classes, async code. Real-world is 33 requests written the way people ask, 8 of them in other languages. BigCodeBench counts the 1,037 of its 1,140 tasks whose libraries run offline.

The rest of the suite moved by a few problems either way:

On 239 held-out chat prompts (200 everyday requests, 37 follow-ups and fixes, 2 from a real chat log), each checked by running the answer:

The same open model wrote these prompts and the training data, so they measure the house style as much as general skill. The 2 chat-log prompts still fail.

All models and the hand-read answers are in the GitHub repo.

Use

The model answers in a fixed template and ends its answer with <|end|>. The tokenizer config needs transformers 5 or newer:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "abhash-chakraborty/ophyx-350m-v3"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

prompt = "Write a function `is_palindrome(s)` that ignores case and non-letters."
ids = tok(f"<|user|>\n{prompt}\n<|python|>\n", return_tensors="pt", add_special_tokens=False).input_ids
end = tok.convert_tokens_to_ids("<|end|>")
out = model.generate(ids, max_new_tokens=512, do_sample=False, eos_token_id=end, pad_token_id=end)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

For a follow-up, put the earlier turns in front, each as <|user|>\n{prompt}\n<|python|>\n{answer}\n<|end|>\n.

On transformers 4.x, load the tokenizer with PreTrainedTokenizerFast(tokenizer_file=hf_hub_download(repo, "tokenizer.json")) instead.

GGUF files for llama.cpp are in ophyx-350m-v3-gguf.

Limits

• Python only, and English prompts only. Ask for JavaScript, C++ or Bash and it writes Python.
• It writes short functions and scripts well. Multi-file work and long reasoning are beyond it: it solves 12 of the 42 hands-on tasks, 2 of the 33 real-world ones and 1.2% of LiveCodeBench.
• Some answers open with a module docstring that describes code it then doesn't write, and a few stop right after that docstring. Function docstrings are usually accurate.
• About half its answers carry full docstrings. The DPO model alone documented 236 of 239, but it lost benchmark ground; v3 trades some of that for v2's coding skill.
• Treat its output like any untested code.

Architecture

Qwen3 layout: 28 layers, hidden size 1,024, 16 query heads and 4 key-value heads of size 64, MLP 2,816, tied embeddings, a 32,768-token vocabulary, RoPE base 100,000 and 4,096 tokens of context. Weights are float32.

Licence

Apache-2.0.