mlx-community/Mistral-Large-4-1T-A52B-Text-4bit

This model mlx-community/Mistral-Large-4-1T-A52B-Text-4bit was converted to MLX format from mistralai/Mistral-Large-4-1T-A52B using mlx-lm with ml-explore/mlx-lm#2018. Refer to the original model card for more details on the model.

This is a text-only conversion. The vision encoder and the DSpark draft head are not included.

The FP8 weights were dequantized and then quantized to 4-bit affine with group size 64 (4.5 bits per weight). The router and two indexer projections stay in bf16, as in the original checkpoint. It does not fit in the memory of one 512GB Mac. Peak memory was about 300 GB on each of two Macs in a short generation test with pipeline parallelism.

Tested with mlx-lm 0.32.3.dev13 and MLX 0.32.4.

Use with mlx

pip install -U mlx-lm

Run it with pipeline parallelism over two Macs, for example with the sharded_generate.py example of mlx-lm:

mlx.launch --backend ring --hostfile hosts.json \
    sharded_generate.py --model mlx-community/Mistral-Large-4-1T-A52B-Text-4bit --pipeline --prompt "hello"

See the MLX distributed guide for the host file and the network setup.