QwEmotion-RP-Embed-0.6B-v0.1

QwEmotion-RP is an emotion classification model trained to determine a character emotion base on a single input message.
The model is trained on all 28 emotions that are compatible with SillyTaverns Character Expressions sprites.

The goal is to create a dedicated model for determining emotions that can be loaded purely in RAM and utilizing only the CPU.
To use the model as it was intended, it is recommended to run llamacpp with the arg --device none for pure CPU inference.

Disclaimer: SillyTavern's Character Expressions extension does not yet accept embedding models. For now this is just a demo that can be played around with.

Usage

Prompting Directions:

Queries: Typically you want your quries to be a single bot chat message: the one you need the character sprite decided for.

Documents: You want the documents to each be a single word, all of which are names of emotions.
The following is the list of emotions this model was trained to classify, sorted in alphabetical order:

emotions: [
    "admiration",    "amusement",      "anger",       "annoyance",
    "approval",      "caring",         "confusion",   "curiosity",
    "desire",        "disappointment", "disapproval", "disgust",
    "embarrassment", "excitement",     "fear",        "gratitude",
    "grief",         "joy",            "love",        "nervousness",
    "neutral",       "optimism",       "pride",       "realization",
    "relief",        "remorse",        "sadness",     "surprise"
]

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Jerald\'s chains rattled as he twisted toward the barred window, throwing off questing hands like a dog shaking water. His glare could have cut steel. The women\'s whimpers died against the concrete walls.\n\n"Had enough." The words cracked like a whip. "Your pet died in that alley outside Huntsville, buried him myself. These girls you\'ve hollowed out? Not my problem."\n\nHe surged forward, chains biting into his wrists. The closest girl stumbled back, eyes wide. Good. Let them see.\n\n"You want leverage?" His laugh came harsh, tasting of blood. "Try again. I watched men die screaming on my table, begging for mothers who weren\'t coming. You think this performance scares me?" He spat at the window. "Find a new trick."',
    'disapproval',
    'excitement',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4001, 0.1208],
#         [0.4001, 1.0000, 0.0439],
#         [0.1208, 0.0439, 1.0000]])

Out-of-Scope Use

The model is trained specifically on emotion classification, and is unlikely to offer any sort of improvement on other tasks.

Bias, Risks and Limitations

Bots that play as more than one character are likely to confuse the model.
It is recommended to use a dedicated narrator bot rather than having main characters also play the role of any side charcters.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: unsloth/Qwen3-Embedding-0.6B
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modalities: Text, Message

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'message': {'method': 'forward', 'method_output_name': 'last_hidden_state', 'format': 'flat'}}, 'module_output_name': 'token_embeddings', 'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'PeftModelForFeatureExtraction'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({})
)

Training Details

This model was finetuned with Unsloth.

based on unsloth/Qwen3-Embedding-0.6B

This is a sentence-transformers model finetuned from unsloth/Qwen3-Embedding-0.6B. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Training Datasets

some-rp-v3 (synthetically augmented)

HF Repo: ToastyPigeon/some-rp-v3

• Size: 499

• Messages: ~10k

• Messages Used: 4096

• Columns: anchor and positive

practical-dreamer-RPGPT_PublicDomain (synthetically augmented)

HF Repo: agentlans/practical-dreamer-RPGPT_PublicDomain

• Size: 3k

• Messages: +128k

• Messages Used: 8192

• Columns: anchor and positive

emotion (minor column augmentation)

HF Repo dair-ai/emotion

• Split: train

• Size: 16k

• Subset Used: 4096

• Columns: anchor and positive

Combined dataset statistics

• Size: 16,384 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 11 tokens mean: 110.47 tokens max: 630 tokens	min: 6 tokens mean: 7.07 tokens max: 9 tokens

• Samples:

  anchor	positive
  eyes widen with concern, her heart racing at the revelation Who might this person be, Mr. Svidrigailov? And what is the nature of the danger they face?	fear
  raises an eyebrow, observing Pangloss's experiment I can't say I've ever seen anything quite like this before. What brings you to such an extraordinary place?	curiosity
  Her breath hitched traitorously as he slicked against her, pleasure blooming where it shouldn't. God. The slow press, deliberate and obscene, had her nails digging half-moons into her own palms. She went still beneath him. Waiting. Pretending she wasn't counting heartbeats until they synchronized. He probably wanted gratitude, or tears, or some theatrical performance of female surrender. "I suppose," she said, voice soft as dust. Not venom. The venom was reserved for Deborah Listing, CDC agent, who'd walked into this trap alone like some rookie with an anthropology degree and hero fantasies. Backup existed. She'd chosen not to use it. Always about the story, wasn't it? The great investigator who needed no partner. The anger tasted metallic, like pennies against her tongue.	anger

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• gradient_accumulation_steps: 2
• learning_rate: 3e-05
• num_train_epochs: 5
• lr_scheduler_type: constant_with_warmup
• warmup_ratio: 0.03
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 3e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.03
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 15.0 minutes

Framework Versions

• Python: 3.11.15
• Sentence Transformers: 5.5.1
• Transformers: 4.56.2
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.3.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}