Built with 🖤 by Nebulixlabs
Horizon-10M-Semantic is an ultra-fast, lightweight semantic embedding model adapted directly from Nebulixlabs/Horizon-10M-Base. While the base model was pre-trained from scratch on a 500M-token mixture, Horizon-10M-Semantic has been further trained on an additional 10M external tokens specifically to convert the causal language model into an effective 320-dimensional semantic representation engine.
The entire Horizon series is built from the ground up with one primary philosophy: blistering speed and near-zero latency. With an estimated semantic accuracy of around ~75%, this model knowingly trades off heavy, state-of-the-art precision in favor of extreme inference speed and minimal memory footprint—making it ideal for real-time applications, CPU-only environments, and edge devices where every millisecond counts.
| Metric | Details |
|---|---|
| 🎯 Estimated Accuracy | ~75% (Moderate — suited for lightweight filtering and fast matching) |
| ⚡ Inference Speed | Ultra-Fast (Sub-millisecond to few-millisecond latency even on standard CPUs) |
| 📐 Embedding Dimension | 320-d dense vectors (compact storage & rapid cosine similarity search) |
| 🪙 Additional Training | +10,000,000 external tokens on top of the 500M-token base checkpoint |
💡 When should you use this model?
If you need state-of-the-art 90%+ retrieval accuracy for complex production RAG, look at larger embedding models (100M–1B+ params). However, if you need instantaneous responses, high-throughput clustering, rapid initial candidate filtering, or local on-device search without a GPU, Horizon-10M-Semantic is engineered specifically for that workflow.
| Component | Value |
|---|---|
| 🔢 Parameters | ~10.06M |
| 🧱 Base Model | Nebulixlabs/Horizon-10M-Base |
| 📚 Vocabulary size | 8,192 (Custom Byte-Level BPE) |
📐 Output Embedding Dimension (d_model) |
320 |
| 🧱 Layers | 6 |
| 👁️ Attention heads | 5 (head_dim = 64) |
| 🔁 MLP hidden size | 864 |
| 🪟 Context length | 128 tokens |
| 🎯 Positional encoding | Rotary (RoPE) |
| ⚖️ Normalization | RMSNorm (pre-norm) |
| 🚀 MLP type | SwiGLU (gated, SiLU activation) |
| 🎲 Dropout | 0.0 |
Inherited from Horizon-10M-Base, pre-trained on a 500M-token mixture:
HuggingFaceFW/fineweb-edu (440M tokens — 88.0%)HuggingFaceTB/finemath (30M tokens — 6.0%)HuggingFaceTB/smollm-corpus (25M tokens — 5.0%)epfml/FineWeb-HQ (5M tokens — 1.0%)To transform the causal base model into a semantic embedding extractor, the model underwent continued training on an additional 10,000,000 external tokens, aligning its 320-dimensional latent space to capture sentence-level semantic similarity and representations.
You can easily generate 320-dimensional semantic embeddings and compute cosine similarity using transformers and torch:
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
repo_id = "Nebulixlabs/Horizon-10M-Semantic"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.eval()
def get_embeddings(texts):
inputs = tokenizer(
texts,
padding=True,
truncation=True,
max_length=128,
return_tensors="pt"
)
with torch.no_grad():
outputs = model(**inputs)
# Mean pooling over token embeddings (accounting for attention mask)
hidden_states = outputs.last_hidden_state if hasattr(outputs, "last_hidden_state") else outputs[0]
attention_mask = inputs["attention_mask"].unsqueeze(-1).expand(hidden_states.size()).float()
sum_embeddings = torch.sum(hidden_states * attention_mask, 1)
sum_mask = torch.clamp(attention_mask.sum(1), min=1e-9)
embeddings = sum_embeddings / sum_mask
# L2 Normalize embeddings for cosine similarity
return F.normalize(embeddings, p=2, dim=1)
sentences = [
"Artificial intelligence is transforming modern software.",
"Machine learning is changing how we build computer programs.",
"I love eating hot pizza on a rainy evening."
]
embeddings = get_embeddings(sentences)
# Compute cosine similarity
sim_1_2 = torch.matmul(embeddings[0], embeddings[1]).item()
sim_1_3 = torch.matmul(embeddings[0], embeddings[2]).item()
print(f"Similarity (AI vs ML): {sim_1_2:.4f}")
print(f"Similarity (AI vs Pizza): {sim_1_3:.4f}")
Note: If your custom model class includes a built-in helper method (e.g., model.encode() or model.get_embeddings()), you can call that directly as well!
@misc{horizon10m_semantic2026,
title = {Horizon-10M-Semantic: An Ultra-Fast Lightweight Semantic Embedding Model},
author = {Nebulixlabs},
year = {2026},
howpublished = {\url{[https://huggingface.co/Nebulixlabs/Horizon-10M-Semantic](https://huggingface.co/Nebulixlabs/Horizon-10M-Semantic)}}
}
model.safetensors 40 MBprojection_head.safetensors 822.1 kBembedding_test_vectors.npz 49.3 kBNebulixlabs/Horizon-10M-Base