Sanskrit-322M-Base

Sanskrit-322M-Base is a 322-million-parameter Sanskrit language model trained from scratch with its own 32,000-token SentencePiece Unigram tokenizer. It is intended for Sanskrit text completion, generation experiments, and further task-specific fine-tuning.

This is a base pretrained model: it predicts the continuation of text. It has not been instruction-tuned or trained as a conversational assistant. It uses a Llama-compatible decoder architecture; its weights were initialized and trained independently, rather than adapted from a pretrained Llama checkpoint.

Published by sraivante under the Apache License 2.0.

Model details

The tokenizer uses identity normalization, byte fallback, and SentencePiece whitespace handling. It does not automatically insert BOS or EOS tokens. Special-token IDs are <unk>=0, <s>=1, </s>=2, and <pad>=3. The original tokenizer.model is included alongside a standard fast tokenizer.json.

Quick start

pip install "transformers>=4.56.2" torch safetensors sentencepiece

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sraivante/Sanskrit-322M-Base"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.float32,
).to(device).eval()

prompt = "धर्मक्षेत्रे कुरुक्षेत्रे"
# Training documents were separated by EOS; use it as a document boundary.
inputs = tokenizer(
    tokenizer.eos_token + prompt,
    return_tensors="pt",
).to(device)

torch.manual_seed(42)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Custom Python model code and trust_remote_code=True are not needed. Keep the prompt plus generated continuation within the 2,048-token context. There is no chat template.

Training data

The packed training mixture contains 1,104,941,200 tokens, including source upsampling. The held-out file contains 5,539,199 tokens. Approximately 0.5% of documents from each source were assigned to validation using a deterministic hash of the document ID.

Preparation included script and language filtering, normalization, deduplication, rule-based cleanup, and targeted OCR correction of the Itihasa portion. The filters aimed to emphasize Sanskrit and reduce other-language material; they do not establish perfect language purity or scholarly accuracy.

The tokenizer was trained independently on a Sanskrit sample using Unigram segmentation and byte fallback. Documents were tokenized and followed by EOS before packing. Batches were sampled as 2,048-token windows.

Dataset names above link to their source repositories. This release includes model artifacts and aggregate data metadata, not copies of the training corpus. Source datasets and underlying works retain their own licenses and terms.

Training procedure

• Hardware: one NVIDIA A100-SXM4-40GB in Google Colab, with checkpoint resumes.
• Precision: BF16 autocast during training; stored parameters in FP32.
• Optimizer: fused AdamW, betas (0.9, 0.95), epsilon 1e-8, weight decay 0.1.
• Learning rate: 3e-4 peak, 700 warmup steps, cosine decay to 3e-5.
• Microbatch: 8 sequences; gradient accumulation: 32 passes.
• Effective update: 256 sequences / 524,288 tokens.
• Gradient clipping: maximum norm 1.0.
• Acceleration: PyTorch scaled dot-product attention and torch.compile.
• Planned training: four token-budget passes, 8,430 optimizer steps.
• Final run: reached step 8,430 and reported completion.
• Published weights: best validation checkpoint from step 8,200.

The best checkpoint had processed approximately 4.299 billion scheduled training tokens. The completed run's nominal budget was approximately 4.420 billion tokens, excluding repeated work after interruptions. Because training samples random windows, these are token-budget passes rather than exhaustive dataset epochs.

Evaluation

The exact best loss is stored inside the supplied best.pt. The other rows come from the completed notebook log.

Validation uses 40 deterministic batches of 8 sequences of 2,048 tokens sampled from the held-out token file, or 655,360 evaluated tokens. It is not a full pass over the validation file. Perplexity is the exponential of the mean cross-entropy and depends on this tokenizer and evaluation procedure; it is not directly comparable to models using other tokenizers or test sets.

No external Sanskrit understanding, translation, instruction-following, or grammaticality benchmarks are reported.

Intended use and limitations

Suitable uses include Sanskrit continuation experiments, studying small language models, and initializing further Sanskrit fine-tuning.

Generated text can repeat phrases, mix genres or languages, contain malformed words, or invent quotations and facts. It should be checked by a knowledgeable reader before use as an educational or scholarly source. The model has not been validated as a translator, grammar checker, factual question-answering system, or general assistant.

The corpus combines classical, encyclopedic, and web sources, so source artifacts, biases, and memorized passages may remain. The model has no instruction-tuning or safety-alignment stage. The 2,048-token context limit is part of the training configuration; longer contexts have not been evaluated.

Export and reproducibility

The original custom PyTorch model was exported into the standard Transformers Llama implementation by splitting packed Q/K/V projections, converting Q/K row order for the RoPE convention, and preserving tied embeddings. The FP32 tensors are retained without quantization.

conversion_report.json records source-file hashes, software versions, tokenizer comparisons, model-logit comparisons, and cached greedy-generation checks. The tokenizer export is checked against the original SentencePiece model rather than relying on a generic Llama tokenizer conversion.

training_data_metadata.json records aggregate source token counts and upsampling factors. training.py contains the original training implementation. convert_to_hf.py contains the conversion and verification procedure.

License

The model weights, tokenizer, and project code in this repository are released under Apache-2.0. See LICENSE. This model's license does not relicense the upstream datasets or underlying texts.