ternlight

A 1.58-bit BitNet-style sentence embedding model distilled from sentence-transformers/all-MiniLM-L6-v2 via quantization-aware training, with post-training int4 quantization at the embedding layer. Weights are ternary {-1, 0, +1}, so inference is adds and subtracts rather than float matmuls, and the whole model — engine, tokenizer, and weights — ships as a single WASM bundle that runs on CPU with no API calls, no GPU, and no runtime download.

ternlight ships in two tiers, same API and same 384-dim output — pick by the size/quality trade. It is designed for short-string semantic similarity (search queries, intent classification, FAQ matching, product cards) deployed on-device (browser, Node, edge runtimes, ARM single-board computers). It is not a frontier model; it trades absolute quality for size and on-device deployability.

Tiers

• base — the quality tier. ~12× smaller on the wire than the fp32 teacher while retaining 0.84 rank-correlation. Use when quality matters more than the last 2 MB.
• mini — the small/fast tier. ~1.6× the throughput at 5 MB, a modest quality step down.

Both tiers: ternary linear weights + int4 embedding table, 384-dim L2-normalized output, 128-token max input, BERT WordPiece vocabulary (30,522, identical to the teacher). Numbers measured on the shipped int4 builds (M-series Mac, Node single-threaded).

How to use

ternlight runs via a custom Rust→WASM inference engine, not via the transformers library. Two paths:

Path 1 — via npm (recommended)

npm install @ternlight/base    # quality tier — 7 MB wire, ~5 ms/embed
npm install @ternlight/mini    # small tier   — 5 MB wire, ~2.5 ms/embed

import { embed, cosineSim, similar } from '@ternlight/base';   // or '@ternlight/mini'

const v1 = embed("arctic terns migrate from pole to pole");
const v2 = embed("longest migration in the animal kingdom");
cosineSim(v1, v2);   // ~0.71 — semantically related, different wording

// Nearest-neighbor search over a corpus
similar("which seabird travels farthest", corpus, { topK: 5 });

The model and tokenizer are bundled into each npm package — no separate download. Works in Node ≥ 18, browsers (via any bundler), Cloudflare Workers, Vercel Edge, Deno, and Bun.

Path 2 — direct download

from huggingface_hub import hf_hub_download

# pick a tier (adjust the paths to match this repo's file layout)
model_bin = hf_hub_download(repo_id="wenshutang/ternlight", filename="base/model-int4.bin")
tokenizer = hf_hub_download(repo_id="wenshutang/ternlight", filename="tokenizer.json")

The .bin files are a custom BitNet b1.58 format. See the engine source for the binary layout and reference forward pass if you want to implement a custom loader in another language or runtime — there is no transformers.AutoModel.from_pretrained() path.

Model details

Training

Distilled from sentence-transformers/all-MiniLM-L6-v2:

1. Distillation objective — cosine/MSE loss between student and teacher 384-dim embeddings over ~1M sentence pairs (search queries, paraphrases, statements).
2. BitNet b1.58 quantization-aware training — all linear layers use ternary weights trained end-to-end with the straight-through estimator. Training under the quantization constraint from the start (rather than quantizing post-hoc) is what preserves fidelity: a naive post-training ternary quant of the same encoder drops sharply, so nearly all of the retained quality comes from the QAT, not the format.
3. Post-training int4 quantization — applied to the token embedding table after QAT, chosen via an ablation over int8/int4/ternary on the table. The embedding table dominates parameter count, so compressing it gives the largest size win for the smallest quality cost.

Training data (base): mix_v3_1M — ~1M pairs from MS MARCO, Quora duplicates, AllNLI, GooAQ, and StackExchange duplicates (English-only, seed=42). See configs/mix-v3-robust.yaml.

Provenance

base (model-int4.bin)

mini (model-int4.bin) — ⚠️ verify against the currently shipped @ternlight/mini build; this sidecar predates the 2026-07-05 package rebuild.

Each .bin ships with a .bin.json sidecar containing full provenance for reproducibility.

Evaluation

Fidelity — Spearman rank correlation vs teacher. Held-out queries, 1,000 deterministic random pairs, seed=42. Spearman of 1.0 = ranks pair similarities identically to the teacher.

Retrieval — SciFact NDCG@10. Absolute retrieval quality on the BEIR SciFact task.

Full methodology and reproduction scripts: eval/quality/RESULTS.md.

Intended use

Designed for:

• Short-string semantic similarity (queries, intents, FAQs, product titles, tags)
• On-device deployment — browsers, Node services, Cloudflare Workers, Deno Deploy, Vercel Edge, Raspberry Pi-class ARM single-board computers
• Cost-free embedding at any scale (no per-call API charges)
• Privacy-sensitive workloads where queries cannot leave the user's device

Not designed for:

• Long-document understanding (max input is 128 tokens — silently truncated above)
• Multilingual workloads (English-only, inherited from MiniLM-L6)
• Maximum absolute quality (use a frontier model like text-embedding-3-large or voyage-3 if quality dominates over size and deployability)

Limitations

• English-only: teacher, tokenizer (bert-base-uncased, no CJK vocab), and training data are English. Non-English text will not tokenize or embed sensibly. Multilingual is the most-requested feature and the pipeline is language-agnostic, but it is not done yet.
• 128-token cap: text longer than 128 BERT WordPiece tokens is silently truncated. Embed at sentence or short-paragraph granularity, not full document.
• Custom runtime required: no transformers path. Use the npm packages or implement a custom loader from the binary format.
• Inherited biases: distilled from all-MiniLM-L6-v2; the same demographic and topical bias caveats from the sentence-transformers corpus apply.
• v0.1: the binary format and JS API may change before v1.0.

License

MIT, matching the teacher model and the ternlight project. See LICENSE.

Citation

@software{ternlight2026,
  title  = {ternlight: a 1.58-bit BitNet sentence embedder in a few MB of WASM},
  author = {Tang, Wen Shu},
  year   = {2026},
  url    = {https://github.com/soycaporal/ternlight}
}

ternlight builds on:

• BitNet b1.58 (Ma et al., 2024) — ternary weight training
• bitlinear by @schneiderkamplab — the reference PyTorch implementation of BitLinear, used directly during training (bitlinear==2.4.6); the Rust inference engine mirrors its forward-pass math byte-for-byte
• sentence-transformers/all-MiniLM-L6-v2 — teacher model

Links

• GitHub: https://github.com/soycaporal/ternlight
• Live demo: https://ternlight-demo.vercel.app
• npm: @ternlight/base · @ternlight/mini