rsi-jev-v6.1-vl-4b-mlx-4bit

RSI-Jev v6.1-VL 4B for Apple silicon, in MLX format, 4-bit. Same API, same effort levels, images included. It is 3.87 GB against 6.33 GB for the 8-bit build, and it gives up some quality for that: the 8-bit build is the lossless choice. Use this one when memory is tight.

pip install "rsi-jev[mlx,vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev"
rsi-jev serve v6.1-vl-4b-mlx-4bit --backend mlx

from serve.decider import Decider
d = Decider("v6.1-vl-4b-mlx-4bit", backend="mlx")
d.decide("I was charged twice. Please refund.",
         {"refund": {"type": "noul", "instructions": "Does the user request a refund?"}})

What it costs

Against the bf16 release, on the same rows and the same GPU type:

The loss sits in knowledge (−0.5) and language (−1.4); the other areas move within noise. About one answer in fifty differs from the bf16 release, mostly where the release's top two options were close. A second 4-bit build made the same way (a different GPTQ run) read 50.73 against 51.35 on another GPU type (−0.62), so treat −0.25 as one draw from a range of roughly −0.3 to −0.6.

What is quantized

The 4-bit codes come from GPTQ (Frantar et al., 2022), as implemented for MLX's affine format, calibrated on 402 development questions (338 text, 64 with images) that are in no evaluation set. The vision tower stays bf16: 4 bits there costs about 7 points of image agreement for 0.5 GB.

Why this layout. Quantizing the embeddings to 8 bits and storing the heads in bf16 saves 0.9 GB over a build that quantizes only the decoder layers. The rule for shipping it was fixed before any number was read: no more than 0.15 below the decoder-only 4-bit build on the Decision Index (same GPU type), and no more than 0.3 points less agreement at any exit. It read 50.99 against 50.94, and its agreement moved by −0.27 / +0.05 / +0.14 points at layers 16 / 20 / 32, so it ships.

How it was checked

• The stored codes are the measured ones. MLX's own mx.dequantize of every one of the 248 quantized matrices equals the weights the numbers above were measured with, bit for bit, and the 8-bit embeddings equal mx.quantize of the release's.
• MLX itself. The build was loaded with the MLX backend and run on the same 2,200 evaluation questions plus 300 held-out questions. Same answer as the bf16 release (served by PyTorch) on 97.1 / 97.9 / 97.9% at layers 16 / 20 / 32 and 98.0% with auto; on the held-out questions 98.7 / 99.0 / 99.7%.
• Effort and serving. low, medium and high give the layer-16, 20 and 32 answers; auto stops where the offline cascade stops. rsi-jev serve --backend mlx answered a text request and an image request.
• Apple silicon. Not yet measured on a Mac. Both builds were checked with MLX itself (CUDA backend) against the PyTorch release; speed and memory on Apple silicon will be added here after a run of scripts/mac_check.sh. On MLX-CUDA the 8-bit build used about 7 GB at 1k input tokens, 12.7 GB at 8k and 26 GB at 32k; the 4-bit build less.

Files

Limitations

• About one answer in fifty differs from the bf16 release, and accuracy at layer 32 is about half a point lower. If that matters, use the 8-bit build.
• Not yet run on Apple silicon. The checks above used MLX's CUDA backend; Metal kernels can round slightly differently, and speed and memory on a Mac are unmeasured.
• Image requests use the same image processor as the PyTorch release when torchvision is installed (rsi-jev[vision]); without it, pixels are resized by PIL and differ slightly.
• Everything else, from the training data and its licences to calibration and the exit policy, is the release's: see the v6.1-VL record.