AnveshAI/AnveshAI-CoderLM-0.1B

---

CoderLM is a compact 100.2M-parameter language model purpose-built for multi-language code generation across 13 programming and markup languages — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on the custom AnvArc (AnveshAI Architecture) — a 15-layer hybrid architecture combining 10× ParallelMinGRU linear recurrence and 5× Differential Grouped-Query Attention (DifferentialGQA) — CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling with an expanded 2,048-token context window.

Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware. At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.

---

Highlights

• 100.2M parameters (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
• ~191MB model size (FP16 model.safetensors) — fits easily in RAM on edge devices and consumer hardware
• AnvArc Hybrid Architecture — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
• 4× Extended Context (2,048 tokens) — trained on full 2K sequences with stable perplexity scaling across all lengths
• 13 languages supported — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
• 18.7% Loss Reduction over 10K Baseline — weighted multi-language loss improved from 1.1637 down to 0.9463 (Perplexity: 2.58)
• Isolated-newline 16K BPE tokenizer — strict indentation and newline preservation (Split(Regex(r"\n"), behavior="isolated"))
• Task & language-oriented special tokens — routing with <|lang:xyz|> tokens and Fill-in-the-Middle (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>)

---

Model Details

---

Comparison: New (2K Context, Step 15,000) vs Old (512 Context, Step 10,000)

The table below compares the new Step 15,000 (2K Context) release against the previous Step 10,000 (512 Context) model across all canonical benchmarks:

1. Overall Model Comparison

---

2. Per-Language Breakdown Comparison (13 Languages)

---

3. Context Length Scaling Curve (New 2K Model)

The new AnvArc architecture demonstrates stable perplexity across sequence lengths up to the full 2,048-token context window:

---

Quick Start

Installation

pip install transformers torch safetensors

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "AnveshAI/AnveshAI-CoderLM-0.1B"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float16,
    device_map="auto",
)
model.eval()

# Prompt with language prefix
prompt = "<|lang:python|>def is_even(n: int) -> bool:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=False))

Fill-in-the-Middle (FIM) Usage

# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = (
    "<|fim_prefix|>def compute_mean(values):\n"
    "    if not values:\n"
    "        return 0.0\n"
    "<|fim_suffix|>\n"
    "    return total / len(values)\n"
    "<|fim_middle|>"
)

inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(output[0]))

---

Special Tokens

CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:

Prompt Formats

Standard Generation:

<|lang:python|>def fibonacci(n):

Fill-in-the-Middle (FIM):

<|fim_prefix|>prefix_code<|fim_suffix|>suffix_code<|fim_middle|>

---

Tips for Best Results

1. Prefix with <|lang:xyz|> — always supply the appropriate language token to activate language-specific representations in AnvArc
2. Utilize Long Prompts up to 2,048 Tokens — unlike the 512 baseline, multi-function context and long file headers are fully supported
3. Use greedy decoding (do_sample=False) for algorithms — produces clean, syntax-valid completions
4. Isolated-newline whitespace — indentation is tokenized into isolated newlines followed by 4-space indent tokens
5. Direct function signature prompts work best — e.g., <|lang:c|>void reverse_string(char *s) {\n

---

Limitations

• 100M scale — this is a lightweight edge model; complex multi-file architectural reasoning is beyond its capacity
• 2,048 token context window — designed for single files, functions, and FIM completions rather than massive codebases
• Base code model — trained for code completion and FIM, not multi-turn conversational chat
• Syntax sensitivity — relies on language routing tokens for optimal formatting

---

Architecture & Training Details

• Base Architecture: AnvArc (15-Layer Hybrid: 10× ParallelMinGRU + 5× DifferentialGQA)
• Attention Mechanism: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
• Recurrent Blocks: ParallelMinGRU with 1-token shift mixing and EOS/language document boundary reset
• Feed-Forward: SwiGLU with $d_{ff} = 1808$
• Tokenizer: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
• Optimizer: AdamW + Muon with Warmup-Stable-Decay (WSD) learning rate schedule
• Precision: FP16 mixed precision
• Best Checkpoint: Step 15,000 (selected by lowest 2K validation loss: 1.1459)
• Hardware: Kaggle GPU environment

---

Citation

@misc{anveshai-coderlm-2026,
  title={AnveshAI CoderLM 0.1B},
  author={AnveshAI},
  year={2026},
  url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
}

---

License

This model is released under the Apache 2.0 License.

---

Built by AnveshAI
AnvArc: 15-Layer Hybrid MinGRU + Differential Attention Code Engine