bonsai: depth12/experts80
Pruned and distilled from Qwen3-30B-A3B-Base for TypeScript code completion. Part of the bonsai project, which applies the Minitron structured-pruning and knowledge-distillation recipe (Muralidharan et al. 2024, Sreenivas et al. 2024) to a Mixture-of-Experts model, a pruning axis neither paper covers.
What was pruned
- Depth: 12 of 48 layers removed (36 kept)
- Experts: 80 of 128 routed experts removed per layer (48 kept)
- Result: ~9.46B parameters (31.0% of baseline's ~30.5B)
Training
3000-step schedule (450-step warmup, peak LR 1e-4), raw-file training cache, distilled against the uncorrected teacher. Best-validation checkpoint, ~step 239 -- the final-step checkpoint (step 2999) overfits severely (pass@1 0.0).
Results
HumanEval-TS (TypeScript MultiPL-E), full 159-problem suite, temperature 0.2, top-p 0.95:
| pass@1 | |
|---|---|
| Qwen3-30B-A3B-Base (unpruned) | 0.717 (114/159) |
| This model | 0.302 (48/159) |
42.1% of baseline accuracy at 31.0% of baseline's parameters.
Notes
From an earlier debugging investigation, before the project adopted the 500-step short schedule used by the other candidates in this collection. Combines both depth and expert pruning. Not recommended for deployment; kept for the project's axis-interaction analysis.
Project
Other checkpoints from this sweep: bonsai collection. Code and full writeup: github.com/SartajBhuvaji/bonsai (private repo).