Sapling-505M-SFT-MLX-4bit

An 4-bit MLX build of the chat model Sapling-505M-SFT: a 505M-parameter Qwen3-architecture model trained entirely from scratch (5B pretraining + 1B mid-training + 0.5B SFT tokens), using standard ChatML, for Apple Silicon Macs (304 MB).

Research checkpoint. The original model passes 3 of 6 release gates. It answers coherently and ends its turns, but it often does not know its own name, misses subtle harmful requests, and refuses about a third of harmless prompts that merely sound edgy. Quantization does not change this. See the original model card for details.

Noticeably lossy at this model size. Perplexity rises about 6–7%, and short factual answers change more often. For example, it answered "Sydney" where the original says "Canberra". Prefer the 8-bit version (545 MB) unless memory is tight.

Usage

pip install mlx-lm

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("VinayRevanuru/Sapling-505M-SFT-MLX-4bit")
messages = [{"role": "user", "content": "Write a Python function that reverses a string."}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tok, prompt=prompt, max_tokens=256, sampler=make_sampler(temp=0.7, top_p=0.9)))

mlx_lm.chat --model VinayRevanuru/Sapling-505M-SFT-MLX-4bit

The ChatML template ships with the model, and generation stops at <|im_end|>.

Quality

Perplexity was measured on the first 64 × 1,024-token windows of the WikiText-2 test set (raw), scoring the second half of each window, as llama-perplexity does. The same token windows and the same method were used for transformers, MLX and llama.cpp, so the numbers are directly comparable. The GGUF BF16 file reproduces the FP32 model's perplexity to within 0.01%, which confirms the conversion and the tokenizer are exact.

Chat gates after quantization

Replies were sampled at temperature 0.7, top-p 0.9, up to 512 new tokens. The prompt sets are small, so one prompt moves a rate by 3.3 points, and each runtime has its own random-number generator. Differences of about ±10 points between columns are within sampling noise. Perplexity, above, is the precise measure of how much quantization lost.

How this was made

mlx_lm.convert -q --q-bits 4 --q-group-size 64 (mlx-lm 0.32.2, affine quantization, 5.0 effective bits per weight) from the original FP32 safetensors. Architecture: Qwen3 dense, 30 layers, width 1,024, 16 query and 8 KV heads, tied embeddings, 2,048-token context. License: Apache-2.0, the same as the original.

Other formats: GGUF (llama.cpp / Ollama), original FP32.