UDGAM LABS · Base model, 4K context · 384M parameters

Udgam-384M-Base-4K

The 4,096-token-context version of Udgam-384M-Base, a 384M-parameter English base model trained from scratch by Udgam Labs: the same pretraining plus a short continued-pretraining stage at 4,096 tokens. It is a decoder-only transformer compatible with the Gemma 4 architecture (Gemma4ForCausalLM), so it runs unmodified in transformers, llama.cpp / Ollama / LM Studio (GGUF) and MLX. Udgam-384M-Chat was fine-tuned from this model.

This is a base (pretrained) model: it continues text. It is not instruction-tuned and not safety-tuned, and it has no chat behaviour of its own. For chat and tool calling use Udgam-384M-Chat.

Model family: Udgam-384M-Base (2,048 context) · Udgam-384M-Base-4K · Udgam-384M-Chat · GGUF · MLX · MLX 4-bit · Technical report

How to use

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("udgamlabs/Udgam-384M-Base-4K")
model = AutoModelForCausalLM.from_pretrained("udgamlabs/Udgam-384M-Base-4K")
ids = tok("<bos>The capital of France is", return_tensors="pt", add_special_tokens=False).input_ids  # add <bos> yourself
print(tok.decode(model.generate(ids, max_new_tokens=30, do_sample=False)[0]))

The model was trained with exactly one <bos> at the start of every document. Since 12 October 2026 the tokenizer adds it: tokenizer(text) returns <bos> + text. If you write <bos> yourself, as above, pass add_special_tokens=False, or you get two. The chat template already starts with <bos>.

GGUF (in gguf/: f16 and Q8_0) with llama.cpp: llama-cli -m Udgam-384M-Base-4K-Q8_0.gguf -c 4096 -p "The capital of France is" -n 40. Ollama (raw completion):

FROM ./Udgam-384M-Base-4K-Q8_0.gguf
TEMPLATE """{{ .Prompt }}"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.15
PARAMETER num_ctx 4096

MLX: mlx_lm.generate --model udgamlabs/Udgam-384M-Base-4K --ignore-chat-template --prompt "The capital of France is" (mlx-lm reads the safetensors directly, and the tokenizer adds the one <bos> the model needs; do not type a literal <bos> as well). Without --ignore-chat-template the prompt is wrapped in the chat template and the base model rambles.

Export parity was checked: transformers fp32, llama.cpp GGUF and MLX give the same greedy tokens, and Ollama's greedy output matches transformers.

Evaluation

Compared with the 2K base on the same evaluation text (bits per byte, lower is better; needle = recall a 5-digit code hidden in filler text):

Short-context quality is unchanged and the model can use 4K positions. Chat-format tool-calling probes (greedy): BFCL v3 simple 66%, multiple 75%, parallel 48%, irrelevance 11% (it calls a tool almost always); held-out app-action exact match 24%. Long-prompt parity was checked at ~3,300 and ~3,900 tokens: llama.cpp and Ollama (num_ctx 4096) give the same tokens as transformers.

Where it falls short

• Small model, small token budget: limited world knowledge, frequent factual errors, repetition, date-reasoning errors and buggy code.
• English only. Context 4,096 tokens at most.
• Not instruction-tuned or safety-tuned: it will continue any text, including harmful text. Do not deploy it directly to end users.
• Prompted as a chatbot without fine-tuning, it tends to ramble in a "thinking aloud" style ("Okay, let me…", "Wait, …"). Fine-tuning (as in Udgam-384M-Chat) removes this.
• Parts of the training text were machine-generated; their styles and errors can carry over.

Training

Pretrained from scratch on roughly 8B tokens of licensed public and synthetic text at a sequence length of 2,048, then continued pretraining for a short stage at 4,096 tokens. See the technical report for an overview.

Licence

Apache-2.0 (weights and tokenizer); see LICENSE. The chat template is the official Gemma 4 template (Apache-2.0).

Citation

@misc{udgamlabs2026udgam384m,
  title        = {Udgam-384M: a small on-device tool-calling model trained from scratch},
  author       = {{Udgam Labs}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/udgamlabs/Udgam-384M-Chat}},
  note         = {Technical report: \url{https://udgamlabs.com/reports/udgam-384m}}
}

Contact

Udgam Labs. Please use the Community (discussions) tab of the model page for questions, issues and feedback.

---

About Udgam Labs. Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. udgamlabs.com