Modified SmolLM2 with Bangla Tokenizer Support

This is a modified version of SmolLM2-135M that includes enhanced Bangla (Bengali) tokenizer support by merging tokens from TituLM.

Model Details

Base Model: HuggingFaceTB/SmolLM2-135M
Tokenizer Enhancement: Merged with TituLM Bangla tokenizer
Original Vocabulary Size: 49,152
Enhanced Vocabulary Size: 180,177
Added Tokens: ~131,025 Bangla-specific tokens

Key Features

✅ Full SmolLM2-135M model architecture
✅ Enhanced Bangla tokenization support
✅ Backward compatible with original SmolLM2
✅ Improved performance on Bangla text

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the modified model
model = AutoModelForCausalLM.from_pretrained("rnnandi/modified_smollm")
tokenizer = AutoTokenizer.from_pretrained("rnnandi/modified_smollm")

# Test with Bangla text
text = "আমি বাংলায় গান গাই"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

Training

This model was created by:

Merging TituLM Bangla tokenizer with SmolLM2 tokenizer
Resizing model embeddings to accommodate new vocabulary
Preserving original model weights and architecture

Citation

If you use this model, please cite both the original SmolLM2 and TituLM:

@misc{smollm2,
  title={SmolLM2: A Family of Small Language Models},
  author={HuggingFace Team},
  year={2024},
  url={https://huggingface.co/HuggingFaceTB/SmolLM2-135M}
}

@misc{titulm,
  title={TituLM: A Bangla Language Model},
  author={Hishab Team},
  year={2024},
  url={https://huggingface.co/hishab/titulm-llama-3.2-1b-v2.0}
}

License

This model is released under the Apache 2.0 License, same as the base SmolLM2 model.

Loading…