mdeberta-v3-base-lite

This model was created through vocabulary pruning of the original microsoft/mdeberta-v3-base model while maintaining full quality for Latin and Cyrillic-based languages.

Supported Languages

• Bulgarian
• English
• French
• German
• Russian
• Spanish
• Swahili
• Turkish
• Vietnamese

Usage

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("rustemgareev/mdeberta-v3-base-lite")
model = AutoModel.from_pretrained("rustemgareev/mdeberta-v3-base-lite")

# Example usage
text = "This is an example text in English."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Verify that the model outputs hidden states (embeddings)
print("Success! Output shape (batch_size, sequence_length, hidden_size):")
print(outputs.last_hidden_state.shape)

Performance Evaluation

Size Comparison

VRAM Usage Comparison

Estimated using Hugging Face Accelerate Model Estimator.

Semantic Similarity Comparison

Evaluation Method: Cosine similarity between embeddings of parallel sentences in different languages, using English as reference.

Test Phrases Used:

• English: "Artificial intelligence learns to understand human languages and helps people communicate."
• Bulgarian: "Изкуственият интелект се учи да разбира човешките езици и помага на хората да общуват."
• French: "L'intelligence artificielle apprend à comprendre les langages humains et aide les gens à communiquer."
• German: "Künstliche Intelligenz lernt, menschliche Sprachen zu verstehen und hilft Menschen bei der Kommunikation."
• Russian: "Искусственный интеллект учится понимать человеческие языки и помогает людям общаться."
• Spanish: "La inteligencia artificial aprende a entender los idiomas humanos y ayuda a las personas a comunicarse."
• Swahili: "Akili ya kisasa inajifunza kuelewa lugha za wanadamu na kusaidia watu kuwasiliana."
• Turkish: "Yapay zeka, insan dillerini anlamayı öğrenir ve insanların iletişim kurmasına yardımcı olur."
• Vietnamese: "Trí tuệ nhân tạo học cách hiểu ngôn ngữ con người và giúp mọi người giao tiếp."

Similarity Results:

License

This model is distributed under the MIT License.