huydang-dek21-embedding

huydang-dek21-embedding is a Vietnamese text embedding focused on RAG and production efficiency:

📚 Trained Dataset:
The model was trained on an in-house dataset consisting of approximately 100,000 examples of legal questions and their related contexts.

⚙️ Efficiency: Trained with a Matryoshka loss, allowing embeddings to be truncated with minimal performance loss. This ensures that smaller embeddings are faster to compare, making the model efficient for real-world production use.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Language: vietnamese
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer
import torch
from pyvi import ViTokenizer

# Load the embedding model from Hugging Face.
# This model is based on PhoBERT, so Vietnamese word segmentation
# is recommended before encoding for optimal performance.
model = SentenceTransformer("huyydangg/DEk21_hcmute_embedding")

# Define a legal question (query) and a set of legal provisions (documents)
query = "Điều kiện để kết hôn hợp pháp là gì?"
docs = [
    "Điều 8 Bộ luật Dân sự 2015 quy định về quyền và nghĩa vụ của công dân trong quan hệ gia đình.",
    "Điều 18 Luật Hôn nhân và gia đình 2014 quy định về độ tuổi kết hôn của nam và nữ.",
    "Điều 14 Bộ luật Dân sự 2015 quy định về quyền và nghĩa vụ của cá nhân khi tham gia hợp đồng.",
    "Điều 27 Luật Hôn nhân và gia đình 2014 quy định về các trường hợp không được kết hôn.",
    "Điều 51 Luật Hôn nhân và gia đình 2014 quy định về việc kết hôn giữa công dân Việt Nam và người nước ngoài."
]

# Apply Vietnamese word segmentation to the query.
# PhoBERT was pretrained on segmented Vietnamese text where
# multi-word expressions are joined by underscores.
segmented_query = ViTokenizer.tokenize(query)

# Apply Vietnamese word segmentation to all documents.
segmented_docs = [ViTokenizer.tokenize(doc) for doc in docs]

# Generate embeddings for the query and documents.
query_embedding = model.encode([segmented_query])
doc_embeddings = model.encode(segmented_docs)

# Compute cosine similarity between the query embedding
# and each document embedding.
similarities = torch.nn.functional.cosine_similarity(
    torch.tensor(query_embedding),
    torch.tensor(doc_embeddings)
).flatten()

# Rank documents by similarity score in descending order.
sorted_indices = torch.argsort(similarities, descending=True)
sorted_docs = [docs[idx] for idx in sorted_indices]
sorted_scores = [similarities[idx].item() for idx in sorted_indices]

# Display the ranked documents and their similarity scores.
for doc, score in zip(sorted_docs, sorted_scores):
    print(f"Document: {doc} - Cosine Similarity: {score:.4f}")

Evaluation

Metrics

Information Retrieval

• Datasets: another-symato/VMTEB-Zalo-legel-retrieval-wseg
• Evaluated with InformationRetrievalEvaluator

Citation

You can cite our work as below:

@misc{DEk21_hcmute_embedding,
  title={DEk21_hcmute_embedding: A Vietnamese Text Embedding},
  author={QUANG HUY},
  year={2025},
  publisher={Huggingface},
}

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}