San-BERT-v2: Advanced Sanskrit Language Representation Model

San-BERT-v2 is an enhanced, state-of-the-art transformer encoder for the Sanskrit language (in Devanāgarī script). It builds upon and substantially upgrades the original San-BERT (sampathlonka/San-BERT, arXiv:2304.01894) through large-scale continued Masked Language Modeling (MLM) across an extensive 4.9+ million sentence corpus combining the Kaggle Sanskrit Text Corpus with vast collections of classical Sanskrit literature, philosophical treatises, and scholarly commentaries.

San-BERT-v2 resolves the severe representation collapse (anisotropy) observed in off-the-shelf San-BERT v1, unlocking competitive representation quality on Sanskrit Semantic Textual Similarity (STS) and zero-shot/fine-tuned classification tasks.

---

Model Summary

• Architecture: BERT-Base (BertForMaskedLM, 12 layers, 768 hidden dimensions, 12 attention heads, 109.5M parameters)
• Primary Script: Devanāgarī (native 30,522 WordPiece vocabulary)
• Context Length: 512 tokens
• Pre-training Objective: Masked Language Modeling (MLM, dynamic 15% masking)
• Framework: PyTorch & Hugging Face Transformers

---

Dataset & Training Data Card

San-BERT-v2 was trained on a multi-source, deduplicated Sanskrit Devanāgarī corpus containing over 4.9 million unique clauses and 36.2 million whitespace words (~65.9 million WordPiece tokens).

Data Sources

1. Kaggle Sanskrit Text Corpus (kartikbhatnagar18/sanskrit-text-corpus):
   • Cleaned, normalized, and segmented Devanāgarī text derived from the comprehensive 285-file Sanskrit Text Corpus on Kaggle.
   • Rigorously stripped of HTML/web boilerplate and segmented into syntactic clauses on pūrṇavirāma (। / ॥).
2. Classical & Vedic Sanskrit Literature:
   • Canonical works spanning the Vedic Samhitās, Brāhmaṇas, Āraṇyakas, and Upaniṣads.
   • Itihāsa (Mahābhārata, Rāmāyaṇa) and Purāṇas.
   • Classical poetry (Kāvya), drama (Nāṭaka), and grammatical literature (Vyākaraṇa).
3. Scholarly Commentaries & Philosophical Treatises:
   • Extensive Sanskrit philosophical discourses (Śāstra and Darśana) including Advaita, Viśiṣṭādvaita, Nyāya, and Mīmāṃsā texts.
   • Traditional sentence-by-sentence word expositions (padārtha), multi-layered traditional commentaries, and analytical glosses (ṭippaṇī).

Tokenization Statistics

---

Benchmark Evaluation

San-BERT-v2 has been rigorously evaluated on two standard Sanskrit natural language benchmarks:

1. Semantic Textual Similarity (STS) — Sandhied Register

Evaluated on 575 human-curated Sanskrit sentence pairs across 8 linguistic categories (lexical synonymy, grammatical sutra-vritti parallels, syntactic scrambling, voice transformation, thematic divergence, etc.) using cosine similarity over mean token embeddings:

Key Finding: Off-the-shelf San-BERT v1 suffered severe representation collapse with pairwise embedding cosine identically equal to 1.000. San-BERT-v2 completely cures this degeneration, pushing Spearman ρ from -3.55% to +63.02%, outperforming all subword foundation baselines.

2. Work-Disjoint 5-Class Genre Classification

Evaluated on a strictly work-disjoint 5-class genre classification benchmark (Classes: Veda, Itihāsa–Purāṇa, Śāstra–Darśana, Stotra, Kalpa–Vedāṅga) across 27 unseen test works over 3 learning rates × 3 random seeds:

---

How to Use

1. Masked Language Modeling Pipeline

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="sampathlonka/San-BERT-v2")
results = fill_mask("धर्मक्षेत्रे कुरुक्षेत्रे समवेता [MASK] युयुत्सवः ।")

for res in results:
    print(f"{res['token_str']}: {res['score']:.4f}")

2. Extracting Sanskrit Sentence Embeddings

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("sampathlonka/San-BERT-v2")
model = AutoModel.from_pretrained("sampathlonka/San-BERT-v2")

sentences = [
    "सत्यमेव जयते नानृतम् ।",
    "सत्येन पन्था विततो देवयानः ।"
]

inputs = tokenizer(sentences, return_tensors="pt", padding=True, truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    # Mean pooling over attention mask
    mask = inputs["attention_mask"].unsqueeze(-1)
    embeddings = (outputs.last_hidden_state * mask).sum(dim=1) / mask.sum(dim=1)

# Compute cosine similarity
similarity = torch.nn.functional.cosine_similarity(embeddings[0:1], embeddings[1:2])
print(f"Cosine Similarity: {similarity.item():.4f}")

---

Citation & Acknowledgements

If you use this model or its benchmarks in your work, please cite:

@misc{sanbert_v2_2026,
  author = {Sampath Lonkar},
  title = {San-BERT-v2: Advanced Sanskrit Language Representation Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/sampathlonka/San-BERT-v2}}
}

The base model architecture is adapted from:

@article{lonkar2023sanbert,
  title={San-BERT: A Pre-trained Language Model for Sanskrit},
  author={Lonkar, Sampath and others},
  journal={arXiv preprint arXiv:2304.01894},
  year={2023}
}

and the training data incorporates the Kaggle Sanskrit Text Corpus curated by Kartik Bhatnagar.