LFM2.5-24B-A2B in EXL3, 4.0 bits per weight
These are EXL3 weights of Liquid AI's LFM2.5-24B-A2B at 4.0 bits per weight, 12.64 GB in total. We made them with exllamav3 1.6.0 from Liquid's BF16 release, LiquidAI/LFM2.5-24B-A2B, with exllamav3's built-in calibration. Every layer is at 4.0 bits; the output head, which the model ties to its embedding, is at 6 bits, and the embedding itself stays in BF16. The raw conversion log and quality output are in quality/.
We serve them with TandemLLM, our inference engine for one DGX Spark.
Quality of every size
We measured each size against the BF16 model with exllamav3's model_diff, on 20 rows of 2,048 tokens of two texts: wikitext-2, and chat (conversations from the oasst1 validation split, human-written answers, in the model's chat template). BF16's perplexity is 21.00 on the wikitext rows and 7.43 on the chat rows.
| size | GB | KL wikitext | top-1 wikitext | KL chat | top-1 chat |
|---|---|---|---|---|---|
| 2.5 | 8.20 | 0.463 | 70.4 % | 0.313 | 78.9 % |
| 3.0 | 9.68 | 0.335 | 75.2 % | 0.195 | 83.6 % |
| 3.5 | 11.16 | 0.226 | 79.0 % | 0.136 | 86.7 % |
| 4.0 | 12.64 | 0.201 | 80.7 % | 0.098 | 88.5 % |
| 4.5 | 14.12 | 0.113 | 85.4 % | 0.063 | 91.2 % |
| 6.0 | 18.63 | 0.080 | 88.2 % | 0.039 | 93.4 % |
On chat text, closer to how the model is used, the KL of every size is 0.49 to 0.68 times its wikitext KL.
Speed
Measured in TandemLLM on one DGX Spark: one request, 512 greedy tokens. Chat is plain decoding. Edit and copy are asks whose answer repeats much of the prompt; there TandemLLM's lookup drafter proposes the repeated text and the model checks it, with the same output as plain decoding.
| size | chat tok/s | edit tok/s | copy tok/s |
|---|---|---|---|
| 2.5 | 185 | 351 | 271 |
| 3.0 | 171 | 328 | 414 |
| 3.5 | 151 | 269 | 287 |
| 4.0 | 144 | 291 | 326 |
| 4.5 | 128 | 281 | 339 |
| 6.0 | 105 | 210 | 280 |
| BF16 | 45 | 154 | 158 |
TandemLLM's default is 3.5: the fastest size whose top-1 agreement with BF16 on chat text is at least 85 %.
How to run it
TandemLLM 0.9.0 runs every size (release notes). On a DGX Spark, install it in setup mode, then switch to this size:
curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.9.0/install.sh
bash install.sh --setup --yes
~/TandemLLM/run/ops/use-profile.sh lfm25-exl3-4.0
use-profile.sh downloads the files of this repository at a pinned commit, checks each one against its sha256 and starts the engine on port 8000. It needs no Hugging Face token. All six sizes are in the collection.
The files are in the standard EXL3 format. exllamav3 1.6.0 loads them; we measured them with it. Other EXL3 loaders such as TabbyAPI should read them too. We have not tested those.
License
These weights are a quantization of Liquid AI's model and carry its license, the LFM Open License v1.0, in LICENSE in this repository. Liquid AI's terms apply to every use of them. Everything else about the model is in its model card.