AnveshAI/AnveshAI-CoderLM-0.1B-Preview

---

CoderLM is a compact 100.2M-parameter language model purpose-built for multi-language code generation across 13 programming and markup languages — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on a novel 15-layer hybrid architecture combining 10× ParallelMinGRU linear recurrence and 5× Differential Grouped-Query Attention (DifferentialGQA), CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling.

Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware. At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.

---

Highlights

• 100.2M parameters (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
• ~191MB model size (FP16 model.safetensors) — fits easily in RAM on edge devices and consumer hardware
• Runs on CPU & Edge Devices — no discrete GPU required, works on laptops, phones, edge devices, and embedded systems
• 15-layer hybrid architecture — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
• 13 languages supported — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
• Isolated-newline 16K BPE tokenizer — strict indentation and newline preservation (Split(Regex(r"\n"), behavior="isolated"))
• Task & language-oriented special tokens — routing with <|lang:xyz|> tokens and Fill-in-the-Middle (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>)
• Verified 1.0397 Validation Loss — 75.54% Top-1 Accuracy and 91.13% Top-5 Accuracy on canonical validation data

---

Model Details

---

Benchmark Results

General Multi-Language Validation (Step 10,000)

Per-Language Breakdown (Ranked by Loss)

---

Quick Start

Installation

pip install transformers torch safetensors

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "AnveshAI/AnveshAI-CoderLM-0.1B-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float16,
    device_map="auto",
)
model.eval()

# Prompt with language prefix
prompt = "<|lang:python|>def is_even(n: int) -> bool:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=False))

Fill-in-the-Middle (FIM) Usage

# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = (
    "<|fim_prefix|>def compute_mean(values):\n"
    "    if not values:\n"
    "        return 0.0\n"
    "<|fim_suffix|>\n"
    "    return total / len(values)\n"
    "<|fim_middle|>"
)

inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(output[0]))

---

Special Tokens

CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:

Prompt Formats

Standard Generation:

<|lang:python|>def fibonacci(n):

Fill-in-the-Middle (FIM):

<|fim_prefix|>prefix_code<|fim_suffix|>suffix_code<|fim_middle|>

---

Tips for Best Results

1. Prefix with <|lang:xyz|> — always supply the appropriate language token to activate language-specific representations
2. Keep max_new_tokens under 256 — the model performs best with short-to-medium completions
3. Use greedy decoding (do_sample=False) for algorithms — produces clean, syntax-valid completions
4. Isolated-newline whitespace — indentation is tokenized into isolated newlines followed by 4-space indent tokens
5. Direct function signature prompts work best — e.g., <|lang:c|>void reverse_string(char *s) {\n

---

Limitations

• 100M scale — this is a lightweight edge model; complex multi-file logic is beyond its capacity
• 512 token context window — designed for snippets, functions, and FIM completions rather than long repositories
• Base code model — trained for code completion and FIM, not multi-turn conversational chat
• Syntax sensitivity — relies on language routing tokens for optimal formatting

---

Architecture & Training Details

• Base Architecture: 15-Layer Hybrid (10× ParallelMinGRU + 5× DifferentialGQA)
• Attention Mechanism: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
• Recurrent Blocks: ParallelMinGRU with 1-token shift mixing and EOS document boundary reset
• Feed-Forward: SwiGLU with $d_{ff} = 1808$
• Tokenizer: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
• Optimizer: AdamW with cosine learning rate schedule
• Precision: FP16 mixed precision
• Best Checkpoint: Step 10,000 (selected by lowest validation loss: 1.0397)
• Hardware: Kaggle GPU environment

---

Citation

@misc{anveshai-coderlm-2026,
  title={AnveshAI CoderLM 0.1B},
  author={AnveshAI},
  year={2026},
  url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
}

---

License

This model is released under the Apache 2.0 License.

---

Built by AnveshAI
15-Layer Hybrid MinGRU + Differential Attention Code Engine