nanoswe-192h-261006

A 1.7B-parameter SWE agent model trained from random init in 192 B200-hours (one 8×B200 node, 24.0 h of training wall-clock) in two stages: a short web-text pretraining stage (6.7B tokens of FineWeb-Edu), then 37.8B tokens of multi-turn tool-calling SWE agent trajectories — no instruction tuning, no RL.

Why the Hub says 3B. The checkpoint holds 2.82B parameters, but 1.14B of them are embedding tables: 1.07B in nanochat-style value embeddings and 67M in the input token embedding. Embedding lookups cost almost no compute, so we count the 1.68B scaling parameters (the transformer matrices plus the LM head), the count our scaling laws use.

At $5 per B200-hour that is roughly $960 of compute, the 192 B200-hour budget. It resolves 25.47% of SWE-bench Verified (pass@1 over K=5 samples on the 483-instance cluster-runnable subset), the new record for the 192 B200-hour budget.

This is a research artifact from a compute-scaling study of SWE agents, not a general-purpose assistant. See Limitations.

The 192 B200-hour record

¹ Same weights, its own protocol (the ```bash agent it was trained for, 2,048 tokens/turn, 60 s timeout), K=5 on the same 483 instances, served with the repaired plugin below; the published 11.01% was K=10 on the earlier serving code.

Quickstart

The architecture (NanoChatForCausalLM) is out-of-tree, so it is served through the nanoswe-vllm plugin. Use v0.2.0 or later: it carries two repairs the evaluation depends on (v0.2.0's model file is the implementation this model was evaluated with):

• Sliding windows. 3 of every 4 layers attend to an 8,192-token window (SSSL pattern, the last layer is always full attention). The plugin must build those layers as sliding-window attention; a plugin that serves every layer full-context diverges from the trained model on every turn whose context exceeds 8,192 tokens.
• Compiled decode. The smear gate threads the previous token's embedding through continuous batching; the repaired plugin keeps that state correct under torch.compile/CUDA graphs and with shared-prefix caching.

Do not use nanoswe-vllm v0.1.0: it predates both fixes.

pip install vllm==0.20.1
pip install "nanoswe-vllm @ git+https://github.com/nanosweb/nanoswe-vllm@v0.2.0"   # or the wheel mirrored in this repo
VLLM_PLUGINS=register_nanoswe VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  vllm serve nanoswe/nanoswe-192h-261006 --max-model-len 34816

--max-model-len 34816 (32k context + one 2k turn) and the plugin are exactly how the model was evaluated. The full evaluation protocol (agent, prompts, tools, grading) is eval/run_eval.sh in nanosweb/nanoswe, which serves the same model code.

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "nanoswe/nanoswe-192h-261006",
  "messages": [{"role": "user", "content": "List the files in the current directory."}],
  "temperature": 0.7, "max_tokens": 8192, "skip_special_tokens": false}'

"skip_special_tokens": false matters: tool calls are delimited by special tokens (see Chat format), and vLLM strips special tokens from the returned text by default, which leaves an agent loop with no parseable action.

In-process:

import nanoswe_vllm   # registers the architecture
from vllm import LLM, SamplingParams

llm = LLM("nanoswe/nanoswe-192h-261006", max_model_len=32768)
out = llm.chat([{"role": "user", "content": "Fix the failing test in tests/test_utils.py"}],
               SamplingParams(temperature=0.7, max_tokens=8192, skip_special_tokens=False))
print(out[0].outputs[0].text)

AutoModelForCausalLM is not supported — there is no transformers-native implementation of this architecture. AutoTokenizer.from_pretrained works normally.

Chat format

Nine special tokens, IDs 32759–32767; the template is in tokenizer_config.json and is applied automatically by /v1/chat/completions (tokenizer and template are byte-identical to nanoswe/nanoswe-192h-260812):

<|bos|><|user_start|>{system}\n\n{user}<|user_end|><|assistant_start|>{assistant}<|assistant_end|>

A leading system message is merged into the first user turn. Generation stops at <|assistant_end|> (id 32763).

Tool calls. Each assistant turn is optional reasoning text followed by exactly one tool call, wrapped in <|python_start|>/<|python_end|> (ids 32764/32765; the name is inherited from nanochat, the channel is tool-agnostic):

<|assistant_start|>{reasoning}<|python_start|>{"name": "bash", "arguments": {"command": "..."}}<|python_end|><|assistant_end|>

The two tools are bash and file_editor. The tool result goes back as the next user turn, as plain text in the <returncode>…</returncode>\n<output>…</output> envelope. The first user turn is the bare issue text (no system prompt, no tool schema): the model was trained without one.

Evaluation

The protocol (agent, prompts, tools, sampling, serving code, the 483 instances) is the one the previous record nanoswe/nanoswe-192h-261002 (15.65%) was evaluated with.

All 2,415 evaluation trajectories (with per-sample grades) are public on Harbor Hub: nanoswe-192h-261006-k5.

Training

Compute is measured from the run logs, wall-clock from each stage's first logged step to its last, × 8 GPUs: web stage 2026-09-20 03:35:43 → 05:48:55 (2.220 h), agent stage 2026-09-20 05:54:49 → 2026-09-21 03:41:37 (21.780 h); 24.000 h × 8 = 192.0 B200-hours. The summed step times are 191.70 B200-hours.

Architecture is derived from nanochat and keeps its distinctive pieces: a smear gate (each layer mixes in the previous token's pre-block hidden state), value embeddings on alternating layers, a mid-trunk backout, per-layer residual/x0 lambdas, QK-norm with a 1.2 split scale, parameterless RMSNorm, relu² MLP, logit softcap at 15, untied embeddings, RoPE θ=1e6, and an SSSL window pattern (three 8,192-token sliding-window layers per full-attention layer).

Data. Stage 1: FineWeb-Edu web text from karpathy/fineweb-edu-100b-shuffle (nanochat's pretraining corpus), BOS-separated documents packed into dense 2,048-token rows. Stage 2: the train split of nanoswe/nanoswe-trajs-261002 (the corpus of the previous record): 1,383,822 deduplicated SWE-smith trajectories from the teacher nemotron-3.5-lightning in native tool-call format (bash + file_editor), across 1,133 repositories (swe-smith-v3-lightning 941,901 / swe-smith-lightning 441,921). The dataset's validation split (1,996 trajectories on held-out instances) was used only to measure bits-per-byte (0.1115 at the end of training).

Contamination

The trajectory corpus follows the same strict rule as the previous records: no training trajectory may come from any of the 12 SWE-bench source repositories, forks included — not merely "no eval instance is copied into training". Provenance is recoverable from every row's instance_id, which encodes the upstream owner__repo.

The train and validation splits span 1,133 distinct repositories. None of them is one of SWE-bench's 12 source repositories, nor a fork or mirror of one (GitHub API fork-parent check: 0 hits). Consequently no evaluated SWE-bench instance can appear in the trajectory stage.

The web stage is general web text (FineWeb-Edu), not derived from the SWE-bench repositories. The FineWeb-Edu shards read (shard_00000–00093 of karpathy/fineweb-edu-100b-shuffle; sha256-identical to the Hub files) were scanned against all 500 SWE-bench Verified instances: verbatim problem-statement, hint and gold/test-patch lines and 13-word shingles. No issue text, fix or test of any instance occurs in them; the few hits are generic overlaps (Python docs or tutorials an issue also quotes, a stdlib error message, HTML boilerplate).

Limitations

• Not a chat model. After a short web-text pretraining stage it was trained only on SWE agent trajectories; on general prompts it produces agent-shaped output (tool calls, patches) or degenerates. It has no instruction tuning and no safety tuning of any kind.
• Needs a matching scaffold. Useful output requires an agent loop resembling the training trajectories — the bash + file_editor tool-call interface above, with tool results fed back as user turns and special tokens kept in the returned text.
• Absolute capability is low. 25.47% pass@1 is far below frontier coding models; the point of the artifact is the compute/accuracy scaling point, not the accuracy.
• Serving requires nanoswe-vllm >= 0.2.0, which reads vLLM attention-metadata internals; it is validated on vLLM 0.20.1 only.

License

MIT.