Mistral-7B-Instruct-v0.3 -- Speaker Identification for German Psychotherapy Transcripts (QLoRA)

Status: Gated Repo

This is a QLoRA fine-tuned adapter for mistralai/Mistral-7B-Instruct-v0.3, adapted as a binary sequence classifier to identify the speaker (Therapist or Patient) of a given utterance in a two-person German psychotherapy session transcript, using surrounding dialogue as context. This repository is self-contained: it includes both the base model weights and the LoRA adapter, so no separate download from mistralai/Mistral-7B-Instruct-v0.3 is required.

Model Details

• Base model: mistralai/Mistral-7B-Instruct-v0.3 (7B), loaded with a SequenceClassification head (MistralForSequenceClassification)
• Adaptation method: QLoRA (4-bit NF4 quantization + LoRA); adapter is kept unmerged and applied at load time (not weight-merged), to preserve exact training-time inference numerics.
• Base model weights and adapter are bundled in this repo for self-contained loading.
• LoRA config: r=16, alpha=8, dropout=0.05, target modules q_proj, k_proj, v_proj, o_proj
• Task: Binary text classification -- T (Therapist) vs. P (Patient)
• Language: German
• Labels: {0: "T", 1: "P"}
• License: Apache 2.0 (inherited from base model); adapter weights not currently released

Intended Use

Supports speaker diarization / re-attribution in transcribed two-person psychotherapy sessions, e.g. as a post-processing step after ASR diarization, or to reconstruct speaker turns when diarization metadata is missing or unreliable. The model is trained on an utterance-level, i.e., each sentence will be classified separately.

Not intended for:

• Clinical decision-making, diagnosis, or use as a medical device
• Identification of individual speakers (this model distinguishes roles, not identities)
• Languages other than German, or therapy modalities/settings substantially different from the training data
• Use as a general-purpose text classifier outside the documented prompt format

Training Data

The model was fine-tuned on German two-person psychotherapy transcripts drawn from multiple public sources. The training data contained approximately 1,500 transcripts of psychotherapy sessions.

No training data or verbatim excerpts are included in or redistributed with this repository.

Each training example consists of:

• no_speaker: the target utterance with the speaker label stripped
• dialog: the surrounding conversational context, with the target utterance marked by --

Class weighting [T: 2, P: 1] was applied during training to counteract the therapist-utterance class being under-represented with about 2:1-ratio and to reduce bias toward over-predicting "Patient".

Training Procedure

• Framework: Transformers Trainer (custom subclass) + PEFT + bitsandbytes
• Quantization: 4-bit NF4, double quantization, compute dtype float16
• Epochs: 2 (early stopping on eval_roc_auc, patience 10)
• Batch size: per-device batch size x2 gradient accumulation steps
• Learning rate: 3e-5, 500 warmup steps, weight decay 0.01
• Optimizer: paged_adamw_8bit
• Loss: weighted BCEWithLogitsLoss (one-hot labels, class weights [2, 1])
• Max sequence length: 2048 tokens
• Precision: fp16 mixed precision
• Hardware: 1xH100 (80GB GPU)

Prompt format

Du bist eine sehr intelligente Psychiaterin.
Untersuche den Verlauf einer Therapiesitzung. Bitte ordne die folgende AUSSAGE im
Psychotherapietranskript dem Sprecher zu. Spricht der Therapeut (T) oder der Patient (P)?
Du erhaeltst den KONTEXT der AUSSAGE unten. Die AUSSAGE wird innerhalb des KONTEXTES
markiert ("--").

## AUSSAGE: {utterance}

## KONTEXT:
{context_with_utterance_marked_by_--}

Evaluation

Evaluated on a held-out test set (3,710 labeled utterances):

Per-class breakdown:

The model performs noticeably better on Patient utterances than Therapist utterances, due to the class imbalance.

How to Use

import torch
from transformers import AutoTokenizer, MistralForSequenceClassification, BitsAndBytesConfig
from peft import PeftModel

repo_id = "https://huggingface.co/ChrisLalk/mistral-7b-therapy-speaker-id"  # single private, self-contained repo

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(repo_id, model_max_length=2048)
tokenizer.pad_token = tokenizer.eos_token

base_model = MistralForSequenceClassification.from_pretrained(
    repo_id,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16,
)
base_model.config.pad_token_id = tokenizer.pad_token_id

model = PeftModel.from_pretrained(base_model, repo_id)
model.eval()

prompt = """Du bist eine sehr intelligente Psychiaterin.
Untersuche den Verlauf einer Therapiesitzung. Bitte ordne die folgende AUSSAGE im
Psychotherapietranskript dem Sprecher zu. Spricht der Therapeut (T) oder der Patient (P)?
Du erhaeltst den KONTEXT der AUSSAGE unten. Die AUSSAGE wird innerhalb des KONTEXTES
markiert ("--").

## AUSSAGE: Und wie fuehlen Sie sich dabei, wenn Sie daran zurueckdenken?

## KONTEXT:
...
-- Und wie fuehlen Sie sich dabei, wenn Sie daran zurueckdenken?
...
"""

inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2048).to(model.device)
with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

pred = id2label[int(probs.argmax())]
print(pred, probs.tolist())

Limitations, Risks & Open Items

• Memorization/extraction risk. A loss-gap membership-inference test found no disproportionate fit to training examples vs. held-out data. A prefix-continuation extraction attempt (reconstructing a generative model from the fine-tuned attention weights + the untouched base LM head) found zero verbatim n-gram overlap on 200 train and 200 held-out examples
• Even though it has been trained on various therapeutic approaches and settings, it may not generalize to other therapeutic approaches, dialects, or transcript formats.
• Requires the exact prompt template above; performance on free-form or reformatted inputs is not evaluated.
• Do not use probability scores as a substitute for validation on your own data.

Citation

@misc{[ChrisLalk]2026,
  author = {[Christopher Lalk]},
  title  = {Speaker Identification in German Psychotherapy Transcripts via QLoRA-finetuned Mistral-7B},
  year   = {2026},
}