health-faq-intent-semantic

A small CPU-only FAQ matcher for US health insurance and retirement-product questions (Medicare Advantage, Part D, Medigap, ACA Marketplace, annuities, IUL), plus broker/consumer chat requests.

You give it a user question. It picks one of 2,241 known FAQs and returns that FAQ's stored answer, word for word, along with a one-word intent (definition, eligibility, cost, coverage, navigation, …). If it is not confident, it returns no answer and lists the top-3 FAQ questions so the app can ask "Did you mean …?"

It does not generate text. Every answer it returns is a stored FAQ answer.

How it works

1. Encoder: sentence-transformers/paraphrase-MiniLM-L3-v2, revision 4ca70771. This is the publisher's quantized ONNX file (model_quint8_avx2.onnx), kept frozen and not fine-tuned. Embeddings use mean pooling and L2 normalization, giving 384 dimensions, with a 128-token input limit.
2. Abbreviations: MA, PDP, IUL and ACA are expanded before encoding.
3. Prototypes: each FAQ class has 6 prototype vectors, built from its paraphrases (13,446 vectors in total). The question gets the score of the closest prototype in each class.
4. Accept or reject: an answer is accepted when similarity ≥ 0.725 and the gap to the second-best FAQ is ≥ 0.03. A question that exactly matches a training question goes through direct lookup. Ambiguous questions, and pronoun-only questions like "Is she eligible?", are rejected.

The full model is about 28 MB: encoder 17.5 MB, prototypes 9.4 MB, answer bank and lookup about 1.2 MB.

Usage

pip install numpy onnxruntime tokenizers huggingface_hub

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("sraivante/health-faq-intent-semantic")
sys.path.insert(0, path)
from inference import SemanticFAQ

faq = SemanticFAQ()
r = faq.answer("Explain MA")
if r["matched"]:
    print(r["intent"], "->", r["answer"])
else:
    print("Did you mean:", [c["question"] for c in r["candidates"]])

Or from the command line: python inference.py "What is Medicare Advantage?"

Output fields: matched, intent, unique_intent (<intent>::<faq_id>), faq_id, answer, matched_question, similarity, margin, exact_question, candidates (top-3), milliseconds.

Intent labels

greeting, definition, mechanism, process, eligibility, cost, comparison, coverage, consequence, timing, recommendation, verification, navigation, account, support. Each intent was labelled once per FAQ group, so every paraphrase of a question shares the same intent.

Evaluation

The test set uses question wordings that were held out of fitting. Every normalized test wording was also removed from exact lookup. Sizes: 37,391 fit rows, 1,101 tuning questions, 1,123 test questions.

Limitations

• Out-of-scope questions: about 29% of questions from unseen FAQ groups were wrongly accepted. Check similarity and margin before showing an answer.
• Near-duplicate FAQs: some FAQs say the same thing (for example several "where are my commissions" requests). The margin rule can reject these even at high similarity, and the app then shows "did you mean" suggestions instead of an answer.
• Metrics measure FAQ selection only: they do not measure whether the facts are correct. The stored answers have not been reviewed by experts. Insurance rules change every year and vary by state, so this is not legal, tax or medical advice.
• Language: English only. Hindi/Hinglish has not been tested. Some family-member phrasings and some annuity wordings are matched wrongly.
• Data: the training data is not published. This repo contains only the stored FAQ questions and answers needed for inference.

Files

License

Apache-2.0, matching the upstream encoder.