mlx-community/Mistral-Large-4-1T-A52B-Text-2bit

This model mlx-community/Mistral-Large-4-1T-A52B-Text-2bit was converted to MLX format from mistralai/Mistral-Large-4-1T-A52B using mlx-lm with ml-explore/mlx-lm#2018. Refer to the original model card for more details on the model.

This is a text-only conversion. The vision encoder and the DSpark draft head are not included.

The FP8 weights were dequantized and then quantized to 2-bit affine for the routed experts and 6-bit affine for the other layers, with group size 64 (2.58 bits per weight). The router and two indexer projections stay in bf16, as in the original checkpoint. Peak memory was about 338 GB in a short generation test.

Tested with mlx-lm 0.32.3.dev13 and MLX 0.32.4.

Use with mlx

pip install -U mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Mistral-Large-4-1T-A52B-Text-2bit")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_dict=False,
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)