rsi-jev-v6.1-vl-4b-mlx-8bit
RSI-Jev v6.1-VL 4B for Apple silicon, in MLX format, 8-bit. It is the same model with the same API, effort levels and image support, and it gives the release's answers in 6.33 GB instead of 9.68 GB.
pip install "rsi-jev[mlx,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v6.1-vl-4b-mlx-8bit --backend mlx
from serve.decider import Decider
d = Decider("v6.1-vl-4b-mlx-8bit", backend="mlx")
d.decide("I was charged twice. Please refund.",
{"refund": {"type": "noul", "instructions": "Does the user request a refund?"}})
For a smaller download there is a 4-bit build (3.87 GB). It gives up about half a point of accuracy.
Quality against the bf16 release
| bf16 release | this build | |
|---|---|---|
| Decision Index 0.2.1, 16k-question sample | 51.24 | 51.39 |
| same answer as the bf16 release, layers 16 / 20 / 32 (2,200 questions) | – | 99.1 / 99.5 / 99.3% |
| accuracy at layer 32 (same 2,200 questions) | 83.7% | 83.5% |
Both Decision Index reads ran on the same rows and the same GPU type; +0.15 is within the read's noise. The bf16 release agrees with itself recomputed in fp32 on 99.5 / 99.7 / 99.8% of the same questions, so the remaining differences are at the level of rounding: they fall on questions where the top two options are nearly tied.
What is quantized
| part | stored as | size |
|---|---|---|
| decoder layers 1–32, every linear layer | 8-bit, groups of 64, round to nearest (mx.quantize) |
3.79 GB |
| token embeddings | bf16, as released | 1.27 GB |
| vision tower | bf16, as released | 0.67 GB |
| decision heads at layers 16, 20 and 32, calibration | fp32, as released | 0.60 GB |
Why this layout. A smaller 8-bit layout was tested first: embeddings in 8 bits too and the heads in bf16 (5.44 GB). The rule for shipping it was fixed before any number was read: within 0.3 of the bf16 release on the Decision Index and at least 99% agreement at every exit. It held the Decision Index (51.28) but agreed on 98.95% of questions at layer 16, one question short of 99%. So this build keeps the embeddings and heads as released.
How it was checked
- The stored weights are the measured ones. The codes, scales and biases of all 248
quantized matrices equal
mx.quantizeof the release's bf16 weights, bit for bit; every other tensor is the release's. - MLX itself. The build was loaded with the MLX backend and run on the 2,200 evaluation
questions (text, a 151-option intent set and 300 image questions) plus 300 held-out questions.
Same answer as the bf16 release (served by PyTorch) on 99.1 / 99.5 / 99.3%
at layers 16 / 20 / 32 and 99.5% with
auto; on the held-out questions 100 / 99.7 / 100%. The mean probability difference is 0.004 on text and 0.008 on images. - Effort and serving.
low,mediumandhighgive the layer-16, 20 and 32 answers;autostops where the offline cascade stops.rsi-jev serve --backend mlxanswered a text request and an image request. - Apple silicon. Not yet measured on a Mac. Both builds were checked with MLX itself (CUDA backend) against the PyTorch release; speed and memory on Apple silicon will be added here after a run of
scripts/mac_check.sh. On MLX-CUDA the 8-bit build used about 7 GB at 1k input tokens, 12.7 GB at 8k and 26 GB at 32k; the 4-bit build less.
Files
| file | what it holds |
|---|---|
tower.safetensors |
decoder layers (8-bit codes, bf16 scales and biases) and the token embeddings (bf16) |
visual.safetensors |
the vision tower, bf16 |
scorer.safetensors, aux_scorers.safetensors |
the decision heads at layers 32, 16 and 20, fp32 |
calibration.json, calibration.safetensors |
one temperature per exit, as released |
mlx.json |
how each part is stored, and the size |
config.json, meta.json, tokenizer and image-processor files |
as released |
SHA256SUMS |
a hash for every file |
Limitations
- Not yet run on Apple silicon. The checks above used MLX's CUDA backend; Metal kernels can round slightly differently, and speed and memory on a Mac are unmeasured.
- Image requests use the same image processor as the PyTorch release when
torchvisionis installed (rsi-jev[vision]); without it, pixels are resized by PIL and differ slightly. - Everything else, from the training data and its licences to calibration and the exit policy, is the release's: see the v6.1-VL record.