Mental Health LLaMA 3 8B LoRA

A LoRA adapter fine-tuned on top of LLaMA 3 8B for mental health counseling conversations. Trained using QLoRA (4-bit quantization) with Unsloth for memory-efficient fine-tuning.

GitHub Repository: hasanfaesal/mental-health-llm

Model Details

Component	Details
Base Model	unsloth/llama-3-8b-bnb-4bit
PEFT Method	LoRA
Quantization	QLoRA (4-bit NormalFloat)
LoRA Rank (r)	32
LoRA Alpha	64
LoRA Dropout	0.1
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Max Sequence Length	2048
Adapter Size	~281 MB

Training

Dataset

Amod/mental_health_counseling_conversations -- 3,512 real counseling Q&A pairs from licensed mental health professionals. The data was split 90/10 for training and validation.

Configuration

Parameter	Value
Optimizer	AdamW 8-bit
Learning Rate	1e-4
LR Scheduler	Cosine
Batch Size	1 (effective 8 via gradient accumulation)
Training Steps	100
Warmup Steps	100
Weight Decay	0.01
Precision	bf16
Gradient Checkpointing	Enabled (Unsloth)

Results

Step	Epoch	Training Loss	Gradient Norm	Learning Rate
10	0.025	2.9583	3.240	9.0e-06
30	0.076	2.3345	1.063	2.9e-05
50	0.127	2.1503	0.685	4.9e-05
100	0.253	2.0442	0.790	9.9e-05
Eval	0.253	2.0212	--	--

Framework Versions

PEFT: 0.18.0
TRL: 0.24.0
Transformers: 4.57.3
PyTorch: 2.9.1
Datasets: 4.3.0

Usage

With Unsloth (recommended)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="hasanfaesal/mental-health-llama3-8b-lora",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

FastLanguageModel.for_inference(model)

prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a compassionate and professional mental health assistant. Provide empathetic, supportive, and helpful responses while being mindful of ethical boundaries. Always encourage professional help when appropriate and never provide medical diagnoses.<|eot_id|><|start_header_id|>user<|end_header_id|>

I've been feeling really anxious lately and can't sleep well.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With PEFT (without Unsloth)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
)

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=quantization_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "hasanfaesal/mental-health-llama3-8b-lora")
tokenizer = AutoTokenizer.from_pretrained("hasanfaesal/mental-health-llama3-8b-lora")

Limitations

This model is a research prototype with significant limitations:

Not a substitute for professional care. Do not use this model as a replacement for licensed mental health professionals.
Limited training. Trained for only 100 steps (~25% of 1 epoch). Response quality will vary.
No safety filters. No built-in mechanisms for crisis detection or response.
Bias. Training data comes from specific counseling platforms and may not represent the full diversity of mental health experiences.
Hallucination risk. May generate plausible but clinically inappropriate responses.

License

Code: MIT License
Dataset: RAIL-D (see dataset page for terms)

Citation

If you use this model, please cite the dataset:

@dataset{amod_mental_health_2023,
  title={Mental Health Counseling Conversations},
  author={Amod Sahasrabude},
  year={2023},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/Amod/mental_health_counseling_conversations}
}

hasanfaesal/mental-health-llama3-8b-lora