Nemotron-3.5-Bolt-74B-A9B-MLX-nvidia24c
Mixed-precision MLX quantization of NVIDIA-Nemotron-3.5-Bolt-74B-A9B, produced on Apple Silicon (M3 Max, 64 GB) using sensitivity-based bit allocation.
Quantization Details
| Quantization method | Mixed-precision (sensitivity-based) |
| Bit allocation | 2-bit base (shared experts 4-bit, attention/embeddings 8-bit) |
| Group size | 64 |
| Quant mode | Affine |
| Total size | 26.05 GB |
Benchmarks
All measurements are against the BF16 reference model on held-out calibration data.
Perplexity (8192 tokens, held-out)
| Model | Perplexity | Delta vs BF16 |
|---|---|---|
| BF16 reference | 13.2468 | — |
| nvidia24c | 14.6395 | +1.39 |
KL Divergence & Top-1 Agreement (256 tokens)
| Model | KL/token | Top-1 agreement |
|---|---|---|
| BF16 reference | 0.000000 | 100.0% |
| nvidia24c | 0.120830 | 88.7% |
Sensitivity Analysis
The bit allocation was guided by per-role KL sensitivity measured at 2-bit against the BF16 reference:
| Role | KL contribution | Relative impact |
|---|---|---|
| Shared experts | 66.70 | Dominant |
| Mamba (SSM) | 60.44 | High |
| Latent projections | 49.05 | High |
| Routed experts | 32.01 | Moderate |
| Attention | 7.14 | Low |
| Embeddings | 2.96 | Low |
Key finding: Shared experts — only 2.4% of parameters but running on every token — account for the largest share of quantization error. The shared-expert width curve flattens after 4-bit (4-bit captures 95% of recoverable KL).
Attribution
Quantized by Hermes Agent using sensitivity-based mixed-precision allocation on Apple Silicon.