nanoswe-depth16-260812

A 0.54B-parameter SWE agent model trained from random init in 2.44 B200-hours (8×B200, 18 min wall-clock) on multi-turn SWE agent trajectories only — no web-text pretraining stage, no instruction tuning, no RL.

At $5 per B200-hour that is roughly $12 of compute. It resolves 0.54% of SWE-bench Verified (pass@1 over K=10 samples).

This is a research artifact from a compute-scaling study of SWE agents, not a general-purpose assistant. See Limitations.

The ladder

Quickstart

The architecture (NanoChatForCausalLM) is out-of-tree, so install the vLLM plugin first — after that vllm serve works with no flags and no trust_remote_code:

pip install "nanoswe-vllm @ git+https://github.com/nanosweb/nanoswe-vllm@v0.1.0"
vllm serve nanoswe/nanoswe-depth16-260812 --max-model-len 32768

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "nanoswe/nanoswe-depth16-260812",
  "messages": [{"role": "user", "content": "List the files in the current directory."}],
  "temperature": 0.7, "max_tokens": 2048}'

In-process:

import nanoswe_vllm  # noqa: F401  — registers the architecture
from vllm import LLM, SamplingParams

llm = LLM("nanoswe/nanoswe-depth16-260812", max_model_len=32768)
out = llm.chat([{"role": "user", "content": "Fix the failing test in tests/test_utils.py"}],
               SamplingParams(temperature=0.7, max_tokens=2048))
print(out[0].outputs[0].text)

The plugin wheel is also mirrored in this repo (nanoswe_vllm-0.1.0-py3-none-any.whl) if you would rather not install from git.

AutoModelForCausalLM is not supported — there is no transformers-native implementation of this architecture. AutoTokenizer.from_pretrained works normally.

Chat format

Nine special tokens, IDs 32759–32767; the template is in tokenizer_config.json and is applied automatically by /v1/chat/completions:

<|bos|><|user_start|>{system}\n\n{user}<|user_end|><|assistant_start|>{assistant}<|assistant_end|>

A leading system message is merged into the first user turn. Generation stops at <|assistant_end|> (id 32763).

Evaluation

Sphinx instances are graded with the harness's leaked tox.ini hunk stripped from the submission: the image ships that edit uncommitted, git add -A sweeps it into every patch, and the grader reverse-applies it — which silently scored every sphinx submission as unresolved. Seed-to-seed variation across identical recipes is ≈0.5 pp, so differences below ~1 pp against another single-seed run are not resolved.

Training

Architecture is derived from nanochat and keeps its distinctive pieces: a smear gate (each layer mixes in the previous token's pre-block hidden state), value embeddings on alternating layers, a mid-trunk backout, per-layer residual/x0 lambdas, QK-norm with a 1.2 split scale, parameterless RMSNorm, relu² MLP, logit softcap at 15, untied embeddings, RoPE θ=1e6, and an SSSL window pattern (three short-window layers per full-attention layer).

Data. The tail recipe over nanoswe/nanoswe-trajs-260812: a single pretraining phase on the broad SWE-smith / swe-zero mixture followed by a short annealed tail on the mini-coder-trajs-400k + swe-zero mixture.

Contamination

The corpus is audited against a strict rule: no training trajectory may come from any of the 12 SWE-bench source repositories, forks included — not merely "no eval instance is copied into training". Provenance is recoverable from every row's instance_id, which encodes the upstream owner__repo, with 0 unparseable rows corpus-wide, so a clean scan is genuinely clean rather than unprovable.

A full scan of all 1,587,217 trajectories (10,548 distinct repos) on 2026-06-21 found 4,516 violating rows (1,370 distinct instances) from 7 of the 12 repos; they were purged, leaving 1,582,701 rows with 0 violations. This model was trained on 2026-06-27, i.e. entirely on the cleaned corpus.

Consequently there is no eval-instance overlap either: no row from any SWE-bench source repo survives in training, so no evaluated instance_id can appear.

Limitations

• Not a chat model. It was trained exclusively on SWE agent trajectories from random init; on general prompts it produces agent-shaped output (shell commands, patches) or degenerates. It has no safety tuning of any kind.
• Needs a matching scaffold. Useful output requires an agent loop resembling the training trajectories — a bash-style tool interface with observations fed back as user turns.
• Absolute capability is low. 0.54% pass@1 is far below frontier coding models; the point of the artifact is the compute/accuracy scaling point, not the accuracy.
• Serving requires the plugin, which reads vLLM attention-metadata internals and is validated against vLLM 0.20.x.

License

MIT.