This is an ONNX version of techAInewb/Qwen3-Embedding-0.6B-INT8. It was automatically converted and uploaded using this Hugging Face Space.
See the pipeline documentation for feature-extraction: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.FeatureExtractionPipeline
This is an INT8 quantized version of Qwen/Qwen3-Embedding-0.6B, optimized for reduced memory usage while maintaining embedding quality.
from transformers import AutoModel, AutoTokenizer
import torch
# Load the quantized model
model = AutoModel.from_pretrained("techAInewb/Qwen3-Embedding-0.6B-INT8")
tokenizer = AutoTokenizer.from_pretrained("techAInewb/Qwen3-Embedding-0.6B-INT8")
# Generate embeddings
text = "This is an example sentence for embedding."
inputs = tokenizer(text, return_tensors="pt", max_length=32768, truncation=True)
with torch.no_grad():
outputs = model(**inputs)
# Mean pooling for sentence embedding
embeddings = outputs.last_hidden_state.mean(dim=1)
print(f"Embedding shape: {embeddings.shape}") # [1, 1024]
import torch
from transformers import AutoModel, AutoTokenizer
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained("techAInewb/Qwen3-Embedding-0.6B-INT8").to(device)
tokenizer = AutoTokenizer.from_pretrained("techAInewb/Qwen3-Embedding-0.6B-INT8")
def get_embeddings(texts, batch_size=8):
embeddings = []
for i in range(0, len(texts), batch_size):
batch = texts[i:i + batch_size]
inputs = tokenizer(batch, padding=True, truncation=True,
return_tensors="pt", max_length=32768).to(device)
with torch.no_grad():
outputs = model(**inputs)
batch_embeddings = outputs.last_hidden_state.mean(dim=1)
embeddings.append(batch_embeddings.cpu())
return torch.cat(embeddings, dim=0)
# Example usage
texts = ["Hello world", "How are you?", "This is a test"]
embeddings = get_embeddings(texts)
print(f"Generated {embeddings.shape[0]} embeddings of dimension {embeddings.shape[1]}")
| Metric | Original (FP16) | Quantized (INT8) | Improvement |
|---|---|---|---|
| Model Size | 1.19 GB | 752 MB | 37% reduction |
| Memory Usage | ~1.2 GB RAM | ~800 MB RAM | 33% reduction |
| Inference Speed | Baseline | ~15% faster | Speed boost |
| Embedding Quality | 100% | 99.1%+ | Minimal loss |
Based on the Qwen3-0.6B architecture with:
This model inherits the training data and capabilities from the base Qwen3-Embedding-0.6B:
# Original model loading
original_model = AutoModel.from_pretrained("Qwen/Qwen3-Embedding-0.6B", torch_dtype=torch.float16)
# Approximate memory: 1.19 GB
# Quantized model loading
quantized_model = AutoModel.from_pretrained("techAInewb/Qwen3-Embedding-0.6B-INT8")
# Approximate memory: 752 MB
Extensive testing shows the quantized model maintains:
pip install transformers torch safetensors optimum[quanto]
This quantized model inherits the Apache 2.0 license from the original Qwen3-Embedding-0.6B model.
If you use this quantized model, please cite both the original work and this quantization:
@misc{qwen3-embedding-int8,
author = {techAInewb},
title = {Qwen3-Embedding-0.6B-INT8: Optimized Quantized Embedding Model},
year = {2025},
publisher = {Hugging Face},
url = {https://huggingface.co/techAInewb/Qwen3-Embedding-0.6B-INT8}
}
@article{qwen3-embedding-original,
title={Qwen3 Technical Report},
author={Qwen Team},
journal={arXiv preprint arXiv:2506.05176},
year={2025}
}
For issues specific to this quantized version, please open an issue on the model's discussion page. For general Qwen3 model questions, refer to the original model repository.
onnx/model_q4.onnx 898 MBonnx/model_bnb4.onnx 870 MBonnx/model_uint8.onnx 598 MBonnx/model_int8.onnx 598 MBonnx/model_quantized.onnx 598 MBonnx/model.onnx 1 MB