What's that?
MXFP4_MOE quantization of Vortex5/G4-Midnight-Macaw-26B-A4B
Use case
Running the model on consumer-grade Blackwell GPUs (due to native fp4 support).
Running on other hardware IS POSSIBLE. I lack conclusive data to say if it's a good idea.
Benchmarks
| Measurement | Q4_K_S (imatrix) | MXFP4_MOE (default, 8 exps) | delta | MXFP4_MOE (16 exps) | delta | MXFP4_MOE (24 exps) | delta |
|---|---|---|---|---|---|---|---|
| pp512 | 3067.83 | 3567.72 | +16.29% | 2783.90 | -9.26% | TBD | TBD |
| tg128 | 108 | 89.87 | -16.79% | 76.43 | -29.23% | TBD | TBD |
| pp65536 | 1748.64 | 2219.20 | +26.91% | 1789.84 | +2.36% | TBD | TBD |
| tg1024 | 87.81 | 88.97 | +1.20% | 67.38 | -23.26% | TBD | TBD |
| pp65536 + tg1024 wall time (s) | 49.14 | 41.03 | -16.5% | 51.81 | +5.43% | TBD | TBD |
| hellaswag% (s=42, n=400, greedy) | 56.5% | 61.00% | +7.96% | 64.00% | +13.27% | 63.5% | +12.39% |
| hellaswag (s=42, n=400, greedy) | 226 | 244 | +18 | 256 | +30 | 254 | +28 |
| winogrande% (n=1267, greedy) | 59.51% | 57.38% | -3.58% | 64.17% | +7.83% | 66.45% | +11.66% |
| winogrande (n=1267, greedy) | 754 | 727 | -27 | 813 | +59 | 842 | +88 |
| Size (bytes) | 15464827008 | 14727180640 | -4.77% | 14727180640 | -4.77% | 14727180640 | -4.77% |
tested on: --temp 0 -ngl 999 -b 512 -ub 512 -ctk q8_0 -ctv q8_0
pp65536/tg1024 test for Q4_K_S was run at -ngl 29 instead - it did not fit otherwise.
pp65536/tg1024 test for MXFP4_MOE(16 exps) was run at -ngl 30 instead - it did not fit otherwise.
Conclusion
MXFP4_MOE allows either Q4_K_S intelligence at -16.5% wall time, or higher than Q4_K_S intelligence at the cost of +5.43% wall time (with experts/tok override to 16), while being of smaller file size.
Is it worth it? I don't know. At this point, my particular config isn't compute-limited, but rather bottlenecked by PCI-E. WYSIWYG.
Disclosure
My only contribution is compute. This is not my merge. Have fun.
Cheers
Google - the base model.
Vortex5 - for the merge effort.
Everyone whose finetunes were included in the merge!