leffm-1.7b
leffm — Liquid Efficient Foundation Model — is LFM3-1.7B with two changes, both trained in:
- a cascade FFN that computes 30% fewer FLOPs per token;
- a vocabulary cut to 65,536 tokens (7.4% fewer parameters).
Runs in llama.cpp via styalai/leffm-llamacpp.
| file | format | size |
|---|---|---|
leffm-1.7b-Q8_0.gguf |
GGUF, Q8_0 — recommended | 1.7 GB |
model.safetensors |
Transformers, bf16 | 3.2 GB |
Performance
Batch-1 decode, Q8_0, greedy, best of five. Hardware: NVIDIA L4 (CUDA), Apple M5 (Metal, and CPU with
-t 4). The CUDA row was measured with a build that has -DGGML_CUDA_FA=OFF: FlashAttention changes
the attention kernels only, not the FFN cascade being compared.
| backend | LFM3-1.7B | leffm | speedup |
|---|---|---|---|
| CUDA | 9.01 ms/token | 7.54 ms/token | 1.19x |
| Metal | 15.16 ms/token | 12.47 ms/token | 1.22x |
| CPU | 22.8 ms/token | 25.4 ms/token | 0.90x |
The GPU speedups are smaller than the 30% FLOP reduction because batch-1 decode is bandwidth-bound and the two selections cost real time. On CPU the model stays slower than dense: ggml's int8 matvec already runs at the part's limit, so reading 30% fewer bytes buys nothing there.
Quality
Perplexity on a held-out slice of styal/calib-data (64 sequences, 115,159 bytes), after one matched
continued-pretraining run of 2.67M tokens — the dense control and leffm trained for the same 163 steps:
| model | vocab | FFN | PPL/token | bits/byte |
|---|---|---|---|---|
| stock LFM3-1.7B | 128,000 | dense | 3.392 | 0.4270 |
| dense, after CPT | 128,000 | dense | 2.832 | 0.3639 |
| leffm | 65,536 | cascade | 3.035 | 0.3885 |
Against the dense model trained the same way, leffm costs +0.069 nats (+6.8% bits/byte). The cut tokenizer needs 1.0043x as many tokens for the same text, so bits/byte is the comparable column.
In llama.cpp, on the same weights, the file measures PPL 3.450 with the cascade and 3.174 with the FFN forced dense (batch-1 decode, Q8_0).
Huggingface
You can use the bf16 version direcly with huggingface or use llama cpp form.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("styal/leffm-1.7b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("styal/leffm-1.7b", trust_remote_code=True, device_map="auto")
messages = [
{"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
Install
Two steps: build the fork, download the file. Then run it.
Prerequisites — git, CMake (3.14 or newer), a C++17 compiler. For NVIDIA GPUs: the CUDA toolkit
(12.x or 13.x) with nvcc on PATH. On macOS nothing else is needed; the default build uses Metal.
1. Build the fork
NVIDIA GPU (CUDA) — FlashAttention is built in, and native detects the local GPU:
git clone https://github.com/styalai/leffm-llamacpp && cd leffm-llamacpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-bench
Apple silicon (Metal — the default backend there) or CPU:
git clone https://github.com/styalai/leffm-llamacpp && cd leffm-llamacpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-bench
-DCMAKE_CUDA_ARCHITECTURES=nativecompiles for the GPU in the machine. Set it explicitly only to cross-compile:89Ada (L4, L40S, RTX 40xx),80A100,86A10 / RTX 30xx,90H100.- The second command enables Metal by itself on Apple silicon, and on a machine without a GPU it builds a CPU-only binary — the same flags work everywhere.
A few minutes later build/bin/llama-cli and build/bin/llama-bench are ready.
2. Download the model
hf auth login # this repo is private
hf download styal/leffm-1.7b leffm-1.7b-Q8_0.gguf --local-dir .
3. Run it
./build/bin/llama-cli -m leffm-1.7b-Q8_0.gguf -p "The quick brown fox" -n 64
-ngl 999 forces every layer onto the GPU; the default is auto, which already does that when the
model fits (1.7 GB). -cnv starts a chat instead of a single completion.
4. Benchmark it
./build/bin/llama-bench -m leffm-1.7b-Q8_0.gguf -p 0 -n 32 -r 3 -ngl 999 --no-warmup
The tg32 row is decode speed, pp512 would be prompt processing.
Nothing else to configure
The file carries its own operating point — llama-cli logs
cascade FFN: c = 5120, k = 2304 (model metadata) at load — so there are no environment variables, no
vocabulary file and no --no-repack to remember.
LLAMA_CASCADE=7680,7680reproduces the dense FFN exactly — greedy output identical toLLAMA_CASCADE=0,0on these same weights. That is the arithmetic identity check.LLAMA_CASCADE=0,0runs these weights with the ordinary dense FFN (this is how the two perplexity numbers above were taken).- Batch 1 only: the cascade is a single-token decode path. Prompt processing stays dense.
What is it?
The cascade FFN
A gated FFN computes
$$\mathrm{FFN}(x) = \big(\mathrm{silu}(xW_{\mathrm{gate}}^{T}) \odot (xW_{\mathrm{up}}^{T})\big) W_{\mathrm{down}}$$,
with F = 7,680 units. The gate has to see every unit — it is the router — but the other two
projections do not:
x ──► W_gate all 7,680 units, the router
│
├─ stage 1: keep the top c = 5,120 units by |silu(gate)|
│
├─ W_up rows of those 5,120 units ──► u
└─ stage 2: keep the top k = 2,304 units by |silu(gate) · u|
│
└─ W_down columns of those 2,304 units ──► FFN(x)
So the FFN gathers 5,120 rows and 2,304 columns instead of running all 7,680 twice: 47.2M → 30.9M MACs per token (65.6% of dense). Both selections are exact top-k (a radix select over the float bit pattern), not a threshold, so the unit counts hold every token.
The vocabulary cut
The stock embedding table has 128,000 rows, 125,033 of them real tokens (the rest is padding). This
model keeps 65,536 of them, chosen from styal/calib-data:
- keep every special token and every single-character token (the BPE base alphabet) — these must exist for any text to tokenize;
- rank the rest by frequency on that corpus, ties toward the shorter surface form;
- keep the top 65,536.
The input embedding and the output head are tied and cut together, and the tokenizer in this repo is the cut tokenizer — nothing is remapped at load time. The removed tokens carry 0.41% of the stock model's probability mass.
Licence
LFM Open License v1.0, inherited from LFM3-1.7B — see LICENSE.