harmony-matching-minilm

This is paraphrase-multilingual-MiniLM-L12-v2, the default encoder in Harmony, fine-tuned on 2,351 pairs of questionnaire items that psychologists rated from 0 to 100 for similarity.

I'm publishing it mainly as a record of a negative result. On questions it hasn't seen in training, it matches items no better than the model it started from, and fine-tuning made it insensitive to negation, which breaks Harmony's polarity step. If you want better matching in Harmony right now, the change described under "Recommendation for Harmony" below works better and needs no training.

Code and full results: https://github.com/mccoolsa/harmony-matching-finetune

Evaluation

I used grouped 5-fold cross-validation on the DOXA AI Harmony matching data, so the left-hand question of every test pair was never seen in training. Scores were calibrated with isotonic regression on a separate held-out group of questions. Fine-tuned results are averaged over 3 seeds. MAE is on the 0-100 rating scale and Spearman is the rank correlation with the human ratings within each fold.

Compared with the untrained base model, fine-tuning changed Spearman by -0.004 (95% interval -0.041 to 0.039, resampling questions) and MAE by +0.04 (-0.26 to 0.33). Tuning the number of epochs and the learning rate with nested cross-validation didn't change that. With a random row split the same recipe looks like it helps (MAE 0.46 better, interval 0.05 to 0.90), but only because every test question has already appeared in training.

The weights here were trained once more on all 2,351 pairs with the same settings, so the table describes the recipe rather than these exact weights.

Known issues

The model largely ignores negation. The median cosine between an item and its negated form ("I feel calm" and "I never feel calm") is 0.98, compared with 0.88 for the base model. That is what the ratings reward, since psychologists rate reverse-keyed items as measuring the same thing. Harmony's polarity step (is_negate=True, the default) depends on negation changing the embedding. With the fine-tuned models it flipped the sign of about a third of pairs on a median cosine gap of 0.007, and ranking ended up worse than the shipped model.

The training data is English only. The base model is multilingual, and I haven't tested how much of that survives fine-tuning.

The ratings are noisy. Where two psychologists rated the same pair, they differed by 26.5 points on average, which puts the best achievable MAE around 19. Many pairs where the two items describe different symptoms of the same disorder were still rated as equivalent.

The pairs come from the DOXA AI Harmony matching challenge repository, which doesn't state a licence.

Recommendation for Harmony

On the same evaluation, using all-mpnet-base-v2 (English only) with the polarity step turned off takes Spearman from 0.168 to 0.273 and MAE from 23.94 to 22.21, without any training. I checked this through the harmonydata package itself:

import os
os.environ["HARMONY_SENTENCE_TRANSFORMER_PATH"] = "sentence-transformers/all-mpnet-base-v2"  # before importing harmony
from harmony import create_instrument_from_list, match_instruments

a = create_instrument_from_list(["I feel anxious", "I have trouble falling asleep"])
b = create_instrument_from_list(["I feel nervous", "I wake up too early"])
result = match_instruments([a, b], is_negate=False)

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("conorxyz/harmony-matching-minilm")
emb = model.encode(["I feel anxious", "I feel nervous"], normalize_embeddings=True)
print(emb[0] @ emb[1])

To try it in Harmony, set HARMONY_SENTENCE_TRANSFORMER_PATH="conorxyz/harmony-matching-minilm" before importing harmony and call match_instruments(..., is_negate=False).

Training

CoSENTLoss from sentence-transformers, with the ratings divided by 100. Three epochs, batch size 16, learning rate 2e-5 with linear decay and no warmup, seed 0, on all 2,351 pairs. Training takes under a minute on an RTX 4060 Ti.

I trained with sentence-transformers 6.1.0, but the files in this repo are the base model's own tokenizer and config files with the fine-tuned weights swapped in. Files saved by the newer library don't load in harmonydata 1.0.7, which pins transformers 4.50. Embeddings from the two setups agree to within 1e-5.

Acknowledgements

The data comes from DOXA AI and the Harmony project at Ulster University. The base model is from sentence-transformers.