SmolLM2-135M-Instruct GLQ 4bpw
[!IMPORTANT] Superseded by
xv0y5ncu/SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw. this build predates block-diagonal RHT, so its effective rate is ~10.07 bpw, not 4.0 — the padding was stored as real bits. The replacement is the same nominal rate with no waste.vllm serve xv0y5ncu/SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw --quantization glqThese weights remain available and unchanged, so existing references keep working.
SmolLM2-135M-Instruct quantized using GLQ (Golay-Leech Quantization).
Note on effective bpw: This model was quantized with power-of-2 FHT padding. Effective storage is ~6.4 bpw due to dimensional padding (hidden_size=576 padded to 1024).
Usage
python -m venv .venv
source .venv/bin/activate
pip install 'glq[vllm]'
import glq.hf_integration
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"xv0y5ncu/SmolLM2-135M-Instruct-GLQ-4bpw",
device_map="cuda",
dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained("xv0y5ncu/SmolLM2-135M-Instruct-GLQ-4bpw")
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Requirements
- transformers >= 5.0
- torch >= 2.0
- glq >= 0.2.8 (
pip install glq)
License
Apache 2.0
🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.