HealthInsurance-FAQ-35M

A 35.65M-parameter GPT (decoder-only transformer) trained from scratch on a synthetic US health-insurance FAQ dataset (Medicare A/B/C/D, Medigap, ACA marketplace, annuities, IUL, life insurance, greetings, out-of-scope refusals). Trained on a single Apple M5 laptop GPU in about 65 minutes. Code, data, logs and evaluation are all included.

The architecture is copied from sraivante/tinystories-35m (base). No pretrained weights were used. Training starts from random normal(0, 0.02) init and sees only the FAQ data.

⚠️ General educational information only, not insurance advice. Answers can be wrong, especially for questions outside the training set. The data was generated with an LLM and has not been reviewed by a licensed or compliance professional. Have one approve it before any customer-facing use (CMS marketing rules apply to MA/PDP content).

Quick start

pip install torch tiktoken tokenizers huggingface_hub safetensors numpy
git clone https://huggingface.co/sraivante/healthinsurance-faq-35m && cd healthinsurance-faq-35m
python code/inference.py "What is Medigap?"
python code/inference.py --variant best "who are you"
python code/inference.py                       # interactive

Prompt format: User: {question}\nAssistant:. The answer ends with <|endoftext|>. Single-turn only.

Model

Training

• Data (sraivante/healthinsurance-faq, also in data/): 1,795 train / 94 val Q&A pairs, plus 10 paraphrases per question. That gives 19,745 train documents (1.45M GPT-2 tokens) and 1,034 val documents. Paraphrases stay with their question's split, so no val question or rewording of one leaks into training.
• Document-aware packing: each document is <|endoftext|> User: … Assistant: … <|endoftext|>. Documents are packed into 1024-token rows (~99% fill).
  • Attention mask: same document AND causal. Positions restart at 0 for each document.
  • Padding gets label −100 and attends only to itself.
  • The model learns to emit EOS at the end of each answer.
• Optimizer: AdamW (β 0.9/0.95, wd 0.1). LR 1e-3 → 1e-4 cosine with 50 warmup steps. Grad clip 1.0.
• Batch and budget: effective batch 32 × 1024 tokens (micro-batch 4 × 8 accumulation). 20 epochs = 900 steps. fp32 on Apple MPS, ~7.4k tok/s.
• Init: normal(0, 0.02), residual projections 0.02/√(2·9). No phase-1 pre-pass.

Metrics per evaluation (results/healthinsurance-faq-35m/)

Val QA F1 is SQuAD token F1 of the greedy answer vs the reference answer, on the 94 val questions. Rep4 and Dist-2 are computed on generated tokens.

Val perplexity bottoms out at epoch 4 and then rises as the model memorizes the training answers. The val answers never appear in training, and ~2/3 of the training tokens are answers repeated across paraphrases. Answer quality (QA F1, Dist-2) keeps improving anyway, which is why final/ is the recommended checkpoint for Q&A.

Does it answer correctly? (every question, final/)

comparison/all_questions_report.md and all_questions_answers.jsonl cover every train and val question, in its original wording and one paraphrase each.

It works as an accurate answer bank for in-dataset questions, including reworded ones. On unseen questions it gives a fluent answer that is close to right about a quarter of the time.

Comparison: same data, different 35M design

The same data, packing, batch, epochs and evaluation were also used to train a custom 35.02M design from scratch: 21 layers × 352 dim, RMSNorm, RoPE, an own 10.7k byte-level BPE, pi-digit init and a phase-1 "unique-token cover" pre-pass. Logs are in results/comparison-baseline-21l-rope/. Its weights are not included.

Bits/char is tokenizer-independent, so it can be compared across the two models (their token perplexities cannot). The shallow-and-wide GPT-2 design wins on every quality metric and trains ~2× faster. These are single runs per design, and several factors differ at once (depth, tokenizer, positions, init, dropout, LR). Two notes on the custom model's pi-digit init:

• The literal 0.003 × digit init, which makes every weight positive, stalled at val PPL ~530.
• The zero-mean 0.003 × (digit − 4.5) init was needed to train at all. This matches the reference repo's finding that init scale matters, not its source.

Reproduce

pip install torch numpy tokenizers tiktoken gmpy2 pandas
python -u code/run.py all --variant tinystories   # prepare (CPU) -> train (GPU) -> metrics table -> zip
python -u code/run.py all                         # the 21L RoPE / pi-init comparison model
python -u code/compare.py && python -u code/eval_all.py
python code/chat_ui.py                            # local streaming chat UI on http://localhost:8001

Training checkpoints every 100 steps and every epoch, and resumes automatically (run.py train). code/tests/ checks that the packed loss equals the per-document loss (no cross-document attention) and that resume works.

Files

License

MIT, for code, weights and data.