Lumma-0.6B-Base

A multilingual language model optimized for efficient deployment and English–Indic language understanding.

600M Parameters • 1 Trillion Training Tokens • 12,288 Context Length • Shared KV

Supported Languages

The model is trained on English and a diverse set of Indic languages, including:

English, Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Odia

---

Overview

Lumma-0.6B-Base is a multilingual decoder-only language model trained from scratch on 1 trillion tokens. It is designed for efficient deployment, long-context inference, and strong multilingual performance across English and Indic languages, featuring a compact transformer architecture, memory-efficient attention mechanisms, and an optimized multilingual tokenizer.

---

Key Features

• Trained from scratch on 1 trillion tokens
• 600 million parameter decoder-only Transformer
• Native English and Indic language pretraining
• Shared KV Attention for memory-efficient inference
• 12,288 token context length
• Grouped Query Attention (GQA)
• RMSNorm with QK Normalization
• SwiGLU feed-forward network
• Factorized tied embeddings
• Large multilingual tokenizer optimized for Indic languages

---

[!NOTE] We do not recommend using base language models for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.

Shared KV

Lumma introduces Shared KV, an alternative key-value caching strategy designed to reduce inference memory requirements without significantly impacting model quality.

Instead of computing independent Key and Value projections, both are derived from a shared latent representation. During attention computation, lightweight Key normalization and RoPE transformations are applied dynamically.

This approach reduces KV-cache memory usage by approximately 50%, making Lumma better suited for long-context inference and memory-constrained deployments.

---

KV Cache Modes

Lumma supports two inference modes depending on deployment requirements.

Shared KV

model.config.kv_cache_mode = "shared"

Recommended when memory is the primary bottleneck.

• Approximately 50% lower KV-cache memory
• Slightly higher compute overhead
• Better suited for long-context inference

---

Vanilla KV

model.config.kv_cache_mode = "vanilla"

Recommended for standard deployments.

• Standard KV-cache implementation
• Lower compute overhead
• Maximum compatibility across inference frameworks

---

Benchmark Results

The following results correspond to the released Lumma-0.6B model trained on 1 trillion tokens.

General Benchmarks

---

Multilingual Tokenization

Efficient tokenization is particularly important for multilingual language models.

Lower fertility indicates fewer tokens are required to represent text, improving both training efficiency and inference cost.

---

Usage

!pip install transformers=='5.4.0'


from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "FrontiersMind/Lumma-0.6B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.bfloat16
).eval()

# Memory-efficient mode
model.config.kv_cache_mode = "shared"

# Standard mode
# model.config.kv_cache_mode = "vanilla"

prompt = "The world is a strange place"

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.3,
    top_p=0.95,
    top_k=20,
    repetition_penalty=1.1,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@misc{lumma2026,
  title={Lumma-0.6B},
  author={FrontiersMind},
  year={2026},
  url={https://huggingface.co/FrontiersMind/Lumma-0.6B}
}