MechaEpstein-8000 — 6-bit MLX Quantization
This is a 6-bit uniform quantization of ortegaalfredo/MechaEpstein-8000 for Apple Silicon using the MLX framework.
Model Details
| Detail | Value |
|---|---|
| Base Model | ortegaalfredo/MechaEpstein-8000 |
| Architecture | Qwen3 (Qwen3ForCausalLM) |
| Parameters | 8B (dense) |
| Quantization | 6-bit uniform (6.500 bits per weight) |
| Model Size | ~6.2 GB (2 shards) |
| Context Length | 40,960 tokens |
| Hidden Layers | 36 |
| Hidden Size | 4096 |
| Attention Heads | 32 (8 KV heads, GQA) |
| Vocabulary Size | 151,936 |
| Tied Embeddings | No |
| Framework | mlx_lm |
Reasoning Format
This model uses <think>...</think> tags for reasoning output:
- Thinking mode (default): Model generates a
<think>...</think>block followed by the final response - Non-thinking mode: No thinking block generated (behaves like a standard instruct model)
- Soft switch: Use
/thinkor/no_thinkin the prompt to toggle per-turn
Recommended Inference Parameters
Thinking Mode
| Parameter | Value |
|---|---|
| Temperature | 1.0 |
| Top P | 0.95 |
| Top K | 20 |
| Min P | 0 |
| Max New Tokens | 32,768 (38,912 for complex problems) |
⚠️ Do not use greedy decoding — it causes performance degradation and endless repetitions.
Non-Thinking Mode
| Parameter | Value |
|---|---|
| Temperature | 1.0 |
| Top P | 0.8 |
| Top K | 20 |
| Min P | 0 |
Usage
from mlx_lm import load, generate
model, tokenizer = load("leonsarmiento/MechaEpstein-8000-6bit-mlx")
messages = [{"role": "user", "content": "Hello!"}]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=True
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=4096, temp=0.6, top_p=0.95)
print(response)
Screenshots
Acknowledgements
- Original model: ortegaalfredo/MechaEpstein-8000
- Base architecture: Qwen/Qwen3-8B
- Quantization framework: MLX