DistilWord2-23k

DistilWord2-23k is the second-generation ultra-compact language model (~23k parameters) in the DistilWord series.

Trained on 12,000 clean, low-entropy synthetic words generated by Harley-ml/LargeWord-1.5M, it achieves state-of-the-art sample efficiency and morphology for its size class.

Benchmark Progression

---

Qualitative Generations

---

Quick Inference

import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

repo_id = "Useruser2statsaltalt/DistilWord2-23k"

tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
model.eval()

prompt = "app"
bos = tokenizer.bos_token or ""
inputs = tokenizer(bos + prompt, return_tensors="pt", add_special_tokens=False)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=12,
        do_sample=True,
        temperature=0.80,
        top_p=0.90,
        top_k=35,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

prompt_len = inputs["input_ids"].shape[-1]
completion = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True)
print(f"Generated: {prompt + completion}")
## Architecture Specifications
Architecture: Qwen Causal LM (Micro-scale)
Hidden Layers: 1
Hidden Size: 16
Attention Heads: 1
Intermediate (SwiGLU): 56
Tied Embeddings: True
Unique Parameters: ~23,024