Mixtral-8x7B-Instruct-v0.1 — FP8 Quantized

FP8 quantized version of mistralai/Mixtral-8x7B-Instruct-v0.1, compressed using llmcompressor.

Quantization Details

The MoE routing gate layers are kept in full precision to preserve expert-selection accuracy. The lm_head is kept in full precision following standard practice.

Model Size

Usage

import os
os.environ["FORCE_DISABLE_VISION"] = "1"

from google.colab import userdata
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

HF_TOKEN = userdata.get('HF_TOKEN')
USERNAME = "frankmorales2020"
REPO_ID = f"{USERNAME}/Mixtral-8x7B-Instruct-v0.1-fp8"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID, token=HF_TOKEN)

# Fix: explicitly set pad_token to eos_token
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"   # left-pad for decoder-only models

model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    token=HF_TOKEN,
    device_map="auto",
    dtype=torch.bfloat16,
)

# Fix: set pad_token_id on the model config too
model.config.pad_token_id = tokenizer.eos_token_id

def run_inference(prompt):
    inputs = tokenizer(
        f"[INST] {prompt} [/INST]",
        return_tensors="pt",
        padding=True,
    ).to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,  # silences the warning
        )

    return tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True
    ).strip()

prompts = [
    "Explain mixture of experts architecture in simple terms.",
    "Write a Python function to compute the Fibonacci sequence.",
    "What are the advantages of FP8 quantization over INT8?",
]

for prompt in prompts:
    print(f"\n{'='*60}")
    print(f"PROMPT: {prompt}")
    print(f"{'='*60}")
    print(f"RESPONSE:\n{run_inference(prompt)}")
    print()

Note: FP8 inference requires an NVIDIA GPU with Ada Lovelace (RTX 4000 series), Hopper (H100), or newer architecture. Ampere (A100) supports FP8 storage but not native FP8 compute.

Hardware Requirements

Base Model

• Base: mistralai/Mixtral-8x7B-Instruct-v0.1
• Architecture: Mixtral MoE — 8 experts, top-2 routing, 46.7B total / ~13B active parameters
• License: Apache 2.0

Limitations

• Slight accuracy degradation vs. the BF16 base (typical for FP8 PTQ)
• FP8 compute not available on Volta, Turing, or Ampere GPUs
• Not evaluated on benchmarks post-quantization — use with appropriate validation for production workloads