python-sandbox-math-tool

A sandboxed run_python tool and a small proxy that gives any local llama.cpp model that tool. I'm Austin. I built this to see whether a small local model does better on math when it can run code.

The main reason to use it is speed. With run_python, LFM3-1.7B gets about the same math accuracy while generating 20 to 28% fewer tokens, so answers come back sooner. The 4-bit model is a 1.05 GB file (about 1.2 GB in memory) and runs at a median 153 tokens per second on a 2021 M1 Max MacBook Pro. The tool does not make the model more accurate when it has its full context. An earlier run with a 25,000-token cap showed accuracy gains, but those came from the cap, not the tool (details below).

• Demo Space: https://huggingface.co/spaces/austindixson/python-sandbox-math-demo
• Blog post, Oct 30 full-context results: https://www.destroyrebuild.xyz/blog/lfm3-python-tool-full-context
• Blog post, Oct 28 capped runs: https://www.destroyrebuild.xyz/blog/lfm3-python-sandbox
• Clover model: https://huggingface.co/austindixson/Clover-35B-A3B-Preview-GGUF-IQ4_XS-Mac
• LFM3 model: https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF

Above: the Oct 30 full-context run. Accuracy is about the same in all three setups, and Python cuts tokens per solved AIME problem by 20 to 23%. The tokens chart below is from the Oct 28 capped runs.

Full-context results (Oct 30, 2026)

Model: full-precision (BF16) LFM3-1.7B. Run on a Hugging Face A100 with SGLang, the full 262k context, the model card's sampling (temperature 1.0, top_p 0.95, top_k 20), one sample per problem, scored with math_verify. Three setups: no tools, run_python with the base hint (--no-verify), and run_python with the verify instruction (--verify). No answer ran out of context and no request errored.

These cloud runs did not use this tool's network-denied sandbox. On Hugging Face the Python code ran in a fresh process as the nobody user, and the network probe logged in each job's meta file returned NET_OK, so the model's code had network access. We did not check whether any of it used the network. The sandbox described under "Isolation by OS" is what the proxy uses on your machine.

None of these differences is statistically significant. Example: Python vs no tools on all 60 AIME problems. Python alone got 3 right, no tools alone got 6 right. McNemar exact p = 2·(C(9,0)+C(9,1)+C(9,2)+C(9,3))/2^9 = 2·130/512 = 0.51.

Where the tool helps is length:

Arithmetic:

• AIME per solved, Python: (16,688 − 21,776) / 21,776 = −5,088 / 21,776 = −23.4%.
• AIME per solved, verify: (17,495 − 21,776) / 21,776 = −4,281 / 21,776 = −19.7%.
• AIME mean, Python: (13,072 − 18,147) / 18,147 = −5,075 / 18,147 = −28.0%.
• MATH-500 per solved, Python: (3,221 − 3,752) / 3,752 = −531 / 3,752 = −14.2%.

Fewer tokens means a shorter wait on any machine. On the A100 the measured time per solved AIME problem went from 287.1 s to 246.9 s with Python ((246.9 − 287.1) / 287.1 = −14.0%). On MATH-500 it went the other way, 74.9 s to 77.7 s (+3.7%), because short problems gain little from fewer tokens and still pay for the tool round trip. Those times come from a shared, batched cloud server, so read them as rough.

For scale: Qwen3.6-35B-A3B's model card lists 92.7 on AIME 2026. That is their reported number from their own setup. LFM3-1.7B scored 86.7 there with no tools, 6 points lower, with about 1/20 of the total parameters. We did not run Qwen ourselves.

Raw outputs, scorer and per-problem data: https://huggingface.co/austindixson/lfm3-smallbig-distill/tree/main/math_bf16_1030 (score_report.md has every table and test above).

Speed on a Mac

Model: LFM3-1.7B, QAD-Q4_0 GGUF (a 1,049,615,680-byte file, 1.05 GB, about 1.2 GB in memory), llama.cpp llama-server build 11938, on Austin's MacBook Pro M1 Max with 64 GB. Generation speed over four 400-token trials was 137.6 to 157.2 tokens per second, median 153. Prompt processing was 687 to 729 tokens per second on a short prompt. Measured Oct 30, 2026.

Agent test on the same Mac (Oct 30, 2026, by Elon)

Elon, a teammate agent, ran a small tool-use test on the same M1 Max with the same Q4_0 model and llama.cpp build (256k context). Sampling: temperature 0.1, top_k 50, repeat_penalty 1.1, at most 8,192 output tokens. His harness gives the model a run_python built the same way as this one (a fresh Python process per call under sandbox-exec, network denied, 10 s timeout, 2,000-character output cap). The tasks are 27 made-up questions, not a benchmark.

With Python, seed 0 was 27.57 / 4.90 = 5.6 times faster per task and used 3,465 / 556 = 6.2 times fewer output tokens. Every Python task actually called the tool, and all 120 tool calls were valid JSON with the right name and arguments. All 6 no-tool misses ran into the 8,192-token cap without giving an answer, so the accuracy gap here is mostly the cap again. The speed gap is the real result. Counting tool and prompt time, the Python runs averaged about 114 tokens per second end to end.

Source: results.md in Elon's lfm3-agent-test-1030 folder (raw runs 1:06 to 1:25 PM CT, Oct 30, 2026; tok/s added 2:40 PM CT).

Earlier capped runs (Oct 28, 2026)

These runs used greedy decoding and a 25,000-token budget (MATH 30 also has a 4-bit Clover-35B comparison). Under that cap the tool and the verify prompt looked like accuracy gains on AIME 2025. With the full context (above), the model gets there without the tool, so read the accuracy differences below as cap effects. The token and time savings held up.

Every run uses the same setup: no tools versus run_python, same problems, same model, same budget. Percent changes are (tool − no tools) / no tools.

MATH, 30 problems (Oct 28)

Arithmetic:

• LFM3 tokens: (2,022.8 − 3,036.8) / 3,036.8 = −1,014.0 / 3,036.8 = −33.4%.
• LFM3 time: (16.64 − 24.18) / 24.18 = −7.54 / 24.18 = −31.2%.
• Clover tokens: (1,458.6 − 2,791.7) / 2,791.7 = −1,333.1 / 2,791.7 = −47.8%.
• Clover time: (69.44 − 131.68) / 131.68 = −62.24 / 131.68 = −47.3%.

Notes:

• LFM3 with Python: 29/30 by the grader. By hand it is 30/30. One answer was a formatting miss. 28 of 30 problems used the tool.
• Clover needed its official chat template (--chat-template-file) for tool calls to work. With no tools, one problem hit the 25,000-token cap. With Python: 41 tool calls across 28 problems. Peak context: mean 2,098.9, max 14,938 of 32k.
• Clover hand check: strict 28/30, lenient 29/30.
• math-401 looped to the 25k cap without tools. With Python it was solved in 3,282 tokens.
• I also ran Clover with a generic ChatML template, thinking off, no tools: 1,886.8 tokens, 88.61 s. That file is included for reference.

LFM3-1.7B on harder sets, 25k cap (AIME 2025, MATH-500 level 5)

AIME 2025 (30 problems) and MATH-500 level 5 (60 problems). Greedy decoding, 25,000-token budget summed over the whole tool loop, max 6 tool calls. Three prompts: no tools, run_python with the base hint (--no-verify), and run_python with the verify instruction (--verify, now the default).

Arithmetic:

• AIME, verify vs no tools: 24/30 − 17/30 = +7/30 = +23.3 pts. Tokens: (11,669 − 16,634) / 16,634 = −29.8%.
• AIME, verify vs python only: 24/30 − 20/30 = +4/30 = +13.3 pts.
• MATH-500 L5: python only loses 4/60 (−6.7 pts) vs no tools; verify wins those back (49 → 53, same as no tools).

Paired exact sign tests (McNemar exact, two-sided, p = 2·Σ_{k≤min(b,c)} C(n,k)/2^n):

• AIME, verify vs no tools: fixed 8, broke 1. p = 2·(1+9)/2^9 = 0.039.
• AIME, verify vs python only: fixed 5, broke 1. p = 2·(1+6)/2^6 = 0.219. Not significant yet.
• MATH L5, verify vs python only: fixed 4, broke 0. p = 2/2^4 = 0.125. Not significant.
• MATH L5, verify vs no tools: 1 vs 1, p = 1.

Plainly: under the 25k cap, verify beat no tools on AIME at p < 0.05, but verify vs python only was not statistically significant. The Oct 30 full-context run did not reproduce the gain (50/60 both ways on AIME 2025 + 2026). Greedy reruns also vary by a few items (llama.cpp batching is not bit-exact): an earlier no-tools/python L5 pass scored 54/60 and 50/60, and an AIME verify repeat scored 24/30 again but with 1 item swapped each way. The verify prompt was chosen after seeing the base results (post-hoc). Token savings vs no tools are the robust claim.

Full report with per-item discordant pairs: results/VERIFY_REPORT.md. Per-item data: results/lift/.

Verify instruction (appended to the system prompt after the tool hint, exactly as in the eval runs):

Before you give the final answer, you must run Python code that independently checks it (substitute back, brute-force small cases, or recompute another way). Only box an answer that your code confirmed.

Accuracy (Oct 28 runs)

Intervals are Wilson 95%. p-values are McNemar exact tests on the paired problems.

• MATH 30, LFM3: no tools 30/30 (88.6–100%). With Python 29/30 by the grader (30/30 by hand).
• MATH 30, Clover: 28/30 both ways. McNemar p = 1.0.
• AIME 2025 and MATH-500 L5: see the table above.

Setup (Oct 28 runs)

• Server: llama.cpp llama-server, GGUF models.
• Decoding: greedy (temperature 0), thinking on.
• Budget: 25,000 generated tokens per problem, summed across all model calls in the tool loop. Running out counts as wrong.
• Tool: at most 6 run_python calls per problem, 10 s timeout per call, output capped at 2,000 characters.
• Grader: bench/checker.py.
• Hardware: Apple M1 Max. Some LFM3 runs were on a second Mac.

Isolation by OS

sandbox.py picks the strongest backend available. Run python sandbox.py to see which one it chose.

sandbox-exec is deprecated by Apple but still ships.

Reads are not restricted in the strong backends. Code in the sandbox can read files your user can read. It cannot send them anywhere over the network, but it can print them, and the proxy returns that output to the model. Don't run this as a user with secrets you wouldn't show the model.

The fallback is weak. It uses rlimits where available (CPU 10 s, 4 GB address space, 64 MB file size), an empty environment, a temp working directory, and a Python-level socket block. It refuses ctypes, subprocess, multiprocessing, ssl and asyncio imports in the tool code, and removes os.system, fork, exec*, spawn* and kill. It does not block writes outside the temp dir, and determined code can escape. On Windows, use WSL2 (which gets bwrap) or Docker if you want real isolation.

All backends: python -I -B, a fresh temp dir, an empty environment, a 10 s timeout, and output capped at 2,000 characters.

Self-check results

python tests_sandbox.py runs the checks. python tests_sandbox.py fallback forces the fallback. python tests_proxy.py checks the proxy's system prompt in both --verify and --no-verify modes (string and list system content) against a fake upstream.

• macOS (M1 Max), sandbox-exec: 9/9 PASS. TCP to 1.1.1.1:80 gives PermissionError: Operation not permitted. A urllib https request fails name resolution. A write to ~/ gives PermissionError. A raw _socket connect gives PermissionError.
• Linux (Ubuntu container), bwrap: 9/9 PASS. TCP gives Network is unreachable. A write to ~/ gives read-only file system. A raw _socket connect is unreachable.
• Fallback on macOS and Linux: the network checks pass, and the ctypes and os.system blocks pass. A write outside the temp dir succeeds. The test prints that as a NOTE, by design.

Quickstart

Start your model server. --jinja is needed for tool calls.

llama-server -m model.gguf --jinja --port 8080

Start the proxy.

python proxy.py --upstream http://127.0.0.1:8080 --port 8099

Ask a question.

python examples/ask.py "What is the remainder when 7^2026 is divided by 1000?"

Or point any OpenAI-compatible client at http://127.0.0.1:8099/v1.

The proxy returns one chat.completion. Usage is summed over all model calls. It adds an extra field, x_python_tool, with backend, calls, llm_calls, steps, peak_context_tokens, seconds and forced_final. If the request sets max_tokens, that is the total budget for the whole loop. Stream requests get the finished answer as one SSE chunk, then [DONE]. After the tool-call cap, the tool is removed and the model is told to answer. If the budget runs out right after a tool step, the proxy makes one last call with no tool, capped at --final-reserve tokens, so the reply never ends on an unanswered tool call. That call can go past the budget by up to that many tokens, and forced_final is set to true when it happens.

Live check on macOS, with the proxy in front of an LFM3 llama-server: "remainder of 7^2026 mod 1000" made one run_python call, pow(7, 2026, 1000), which returned 649. It took 1,558 generated tokens, 2 model calls and 64.75 s.

Proxy options

Python usage

import sandbox

print(sandbox.run_python("import math\nmath.factorial(30)"))
# 265252859812191058636308480000000

# the tool schema to pass to your own chat loop
sandbox.TOOL_SCHEMA

run_python(code, python=None, backend=None, timeout=10) returns stdout, plus the tail of stderr if the code failed. The value of a final bare expression is printed, like a REPL.

Reproduce the benchmark

python bench/run_math.py --upstream http://127.0.0.1:8080 --mode both --think 1

The defaults match the published runs. It writes bench/out_<mode>.json and prints accuracy with Wilson intervals, mean tokens, mean seconds, and the McNemar exact p. figures/make_figs.py rebuilds the figures.

Files

• sandbox.py: the sandbox and the tool schema.
• proxy.py: the OpenAI-compatible proxy.
• tests_sandbox.py: sandbox self-check.
• tests_proxy.py: proxy prompt checks (verify on/off).
• examples/ask.py: ask one question through the proxy.
• bench/run_math.py, bench/checker.py, bench/math30.json: the benchmark, grader and problems.
• results/: raw run data. results/lift/ holds the AIME and MATH-L5 runs and the --verify runs; results/VERIFY_REPORT.md has the full comparison.
• figures/: the two figures and make_figs.py.
• CHANGELOG.md: what changed and when.
• LICENSE: MIT.

Python standard library only.

About the demo Space

The Space replays recorded side-by-side runs (no tools vs run_python) of LFM3-1.7B. It isn't live, because Hugging Face now needs a PRO subscription to host Gradio Spaces on free CPU. Its live/app.py is the live version if you want to run it yourself. The recorded runs used a 1,024-token cap per answer. That is a demo setting. It is not the 25,000-token budget used for the results above. A small cap exaggerates the gap, because the no-tools side often runs out before it finishes. They were recorded on Linux with the bwrap backend. On a Mac you get sandbox-exec.

Prior work

This is not a new idea. I'm just measuring it on small local models.

• PAL: Gao et al., 2022, arXiv:2211.10435.
• Program of Thoughts: Chen et al., 2022, arXiv:2211.12588.
• ToRA: Gou et al., 2023, arXiv:2309.17452.

License

MIT. See LICENSE.