Sikia interviewer v8 on Gemma 4 E4B (LoRA adapter, card format 12n)

The same listening adapter as v8 on Gemma 4 E2B, trained on the larger Gemma 4 E4B base with identical data and settings. It reads a respondent's reply during an offline survey interview and returns one short JSON reply. It follows the Sikia offline interview design: the app leads the interview (question order, skip logic, repeat rounds, follow-ups, everything that is spoken) and the model only listens. Each call gets the standing instructions (instructions.txt) and one short card.

Why this adapter exists. It is the "ceiling" run of a base-model comparison: does a larger base read the cards better than E2B? On real bean-trial replies it does. E4B is a research comparison, not the version recommended for phones. v8 on E2B stays the deployment choice until E4B has been measured on a real 8 GB phone (4-bit size, memory and seconds per reply are not yet measured). The base weights are about 23 GB in bf16.

Status: research pilot. Use it together with the app's checks, as for v8. Sikia's relevance test (off-topic replies in realistic interviews) is pending. Don't use it for real data collection yet.

What it does

Behaviour, card format, instructions and the app's checks (slmlib/cards.py, slmlib/controller.py) are the same as for v8 on E2B; see that model card for the details of name lists, free-text names, spoken numbers, negation and "nothing else" replies.

Try it

Accept the Gemma licence on Hugging Face first, and use transformers 5.17 or later (the requirement of the E2B adapter; not separately checked for older versions on E4B).

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

BASE, ADAPTER = "google/gemma-4-E4B-it", "smutuvi/sikia-listener-v8-gemma4-e4b-lora"
proc = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
system = open(hf_hub_download(ADAPTER, "instructions.txt"), encoding="utf-8").read().strip()

card = """LANG: sw
CURRENT: q=village type=select_one "Chagua kijiji:"
OPTIONS: 1=Kagera 2=Misenyi
CONTEXT: -
RESPONDENT SAID: "Tanga." """
msgs = [{"role": "system", "content": system}, {"role": "user", "content": card.strip()}]
prompt = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
enc = proc.tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=80, do_sample=False)
print(proc.tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
# expected for this card: {"act": "clarify", "value": null, "say": ""}   (the name is not in the list)

Cards must match the training format byte for byte. slmlib/cards.py writes every card.

Training data

The same 10,943 cards as v8 (real bean-trial replies, drafted interviews, Sikia practice interviews, real Sikia replies on other forms, and written rows for comments, numbers, name lists, free-text names and review fixes). The data itself is not published (it contains what respondents said). The written rows are synthetic and need a native-speaker check. Nothing comes from Sikia's held-out forms or the forms in Sikia's relevance test.

Evaluation

Scored on the v8 test set (2,597 cards) with eval/score_v12.py, model alone, one seed per base. v8 on E2B and the other bases (Tiny Aya Global, Qwen3.5-2B, LFM2.5-1.2B) were trained on the same data with the same settings. Differences of 1 to 3 cards on the real sets (118 Sikia, about 190 bean) are within run-to-run noise.

Reading it. E4B is the only base tried that beats v8 by more than noise: on real bean it raises trait F1 and cuts wrong saves from 25.4% to 18.0% (about 14 cards of 189). It is a few cards worse on real Sikia right reply type and re-asks, and its free-text word overlap on real Sikia is lower (0.469 against 0.634, a small sample). The gain suggests part of E2B's weakness on real bean multiple-choice cards is capacity. Results for the name-list, free-text-name and review-fix groups are not compared here.

Confidence. Using the probability of the reply the model wrote, the validated cut-off is 0.5 (a wrong save counted as 3 re-asks). On all test cards that gives 3.1 wrong saves and 2.3 re-asks per 100 answers, with expected calibration error 0.022 (v8 on E2B: 2.8, 3.8 and 0.014 at its cut-off of 0.6). Scoring each allowed reply against the others (choice probability) is over-confident here too (calibration error 0.038).

Known weak spots are those of v8: real bean multi-choice answers (options added or dropped, some doubtful keys), negation on single-choice health questions, and free text on real Sikia speech.

Training procedure

LoRA (PEFT) on the language model's linear layers only; the vision and audio towers are untouched. Loss on the JSON reply only. Rank 16, alpha 32, 3 epochs, effective batch 16 (2 × 8), bf16, one GPU, seed 42. Settings in train_config.json.

Limits

Research use. Respondent replies are not in the repository. The adapter holds no respondent text but was trained on real replies; confirm consent and project terms before sharing it widely. The Gemma licence applies to the base model and to this adapter.