Sikia interviewer v8 (LoRA adapter, card format 12n)

A LoRA adapter on stock Gemma 4 E2B that reads a respondent's reply during an offline survey interview and returns one short JSON reply. It follows the Sikia offline interview design: the app leads the interview (question order, skip logic, repeat rounds, follow-ups, everything that is spoken) and the model only listens. Each call gets the standing instructions (instructions.txt) and one short card.

Versions. v8 is trained like v4 and v6c (same base model, card format, instructions and settings) on wider data. It follows v7 (name lists) and adds free-text names, spoken numbers, negation, and "nothing else" replies. v4 remains the version in use until a later version passes Sikia's relevance test.

Status: research pilot. Use it together with the app's checks (below): several of the gains come from the model and the app together. Sikia's relevance test (off-topic replies in realistic interviews) is pending. Don't use it for real data collection yet.

What it does

The app counts the goal's target itself and stops a follow-up when nothing new is said, so it does not use the model's goal_met. Follow-up questions are worded by the app; follow_up stays empty.

What is new since v6c

• Names that are not in the list. On questions whose options are names (regions, districts, villages, schools, markets, groups), v6c picked the nearest option for a name that was not there ("Tanga" became Kagera). v8 says the name is not in the list (clarify), or picks "Other" when the list has it, including for kinds of list it never trained on.
• Free-text names that are not names. "45", "jana" or "ndiyo" as a farmer's name, or a place when a person's name was asked, now get a re-ask.
• Spoken numbers. "elfu mbili na mia nane na nane" is 2808.
• Negation. "sijayaona, iko vizuri tu" to "is there any pest or disease?" is no.
• Yes or no with more said. "Ndiyo, na kuku pia" is yes.
• "Nothing else" in follow-up checks. "hakuna", "sina kingine" give no new point; "hakuna maji ya kutosha" is a point.

The app's checks

The adapter is meant to run with slmlib/cards.py and slmlib/controller.py in the project code. They do four things the model cannot do alone: check a chosen option against the name actually said (ground_choice), check that a free-text answer is the kind of name asked for (check_text), read spoken numbers and overrule a disagreeing number (check_number), and count a follow-up goal's target. Scores below are shown for the model alone and with these checks (eval/score_v12.py --app-check).

Example

LANG: sw
CURRENT: q=top_constraints type=select_multiple "Changamoto kubwa ni zipi?"
OPTIONS: 1=Ukame 2=Wadudu 3=Magonjwa 4=Udongo duni 5=Soko 6=Mbegu duni 7=Hakuna
CONTEXT: -
RESPONDENT SAID: "Wadudu ni wengi, na pia soko halina bei nzuri."

{"act": "answer", "value": "2 5", "say": ""}

Cards must match the training format byte for byte. slmlib/cards.py writes every card and applies the app's post-processing.

Try it

Accept the Gemma licence on Hugging Face first, and use transformers 5.17 or later (earlier 5.x versions compute Gemma 4 E2B differently with the cache on, and the adapter then has no effect).

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

BASE, ADAPTER = "google/gemma-4-E2B-it", "smutuvi/sikia-listener-v8-gemma4-e2b-lora"
proc = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
system = open(hf_hub_download(ADAPTER, "instructions.txt"), encoding="utf-8").read().strip()

card = """LANG: sw
CURRENT: q=village type=select_one "Chagua kijiji:"
OPTIONS: 1=Kagera 2=Misenyi
CONTEXT: -
RESPONDENT SAID: "Tanga." """
msgs = [{"role": "system", "content": system}, {"role": "user", "content": card.strip()}]
prompt = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
enc = proc.tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=80, do_sample=False)
print(proc.tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
# expected: {"act": "clarify", "value": null, "say": ""}   (the name is not in the list)

With the project code, a whole interview on any form, typed or spoken:

python demo/form_interview.py --adapter smutuvi/sikia-listener-v8-gemma4-e2b-lora --form bean_tricot          # terminal
python demo/form_interview.py --adapter smutuvi/sikia-listener-v8-gemma4-e2b-lora --form clinic --lang en
python demo/form_app.py                                                      # browser; lists this adapter

Speech is transcribed separately (the base model with the adapter switched off, or the ndizi speech model): one skill per adapter.

Training data

10,943 cards; the data itself is not published (it contains what respondents said).

The written rows (2,021 in total for v7 and v8) are synthetic and need a native-speaker check; the names in them are invented or common place and person names, none from Sikia's relevance or held-out forms. Nothing comes from Sikia's 4 held-out forms or from the forms in Sikia's relevance test.

Evaluation

Test cards: the v6c test (the same 1,872 cards for every version) plus new groups for names (290), free-text names (233) and the v7 review fixes (202), 2,597 in all. v6c and v7 were scored on the same cards; groups that did not exist for v6c are marked n/a.

Model alone, compared on the same cards:

New groups (v7 against v8; v6c was not scored on them):

The name-list and free-text groups are synthetic and were written to match the rules; the numbers show that the model and the app agree with them, and that the rules carry over to kinds of names not in training. They do not show how well real respondents' replies are handled. Differences of 1 to 3 cards are within run-to-run noise (one seed was trained for v8).

Confidence. eval/confidence.py scores every reply, and eval/confidence_report.py picks a cut-off on the validation cards (a wrong save counted as 3 re-asks) and reports the test. The app saves an answer when the confidence is at or above the cut-off and asks again below it. Per 100 cards whose correct reply is an answer, using the probability of the reply the model wrote (cut-off 0.6):

The cut-off trades lost data for re-asks: on all test cards, 0.8 gives 1.5 wrong saves and 6.4 re-asks, 0.9 gives 0.7 and 9.7. Expected calibration error is 0.014. Of the 1,401 answers with confidence 0.9 or more, 99.2% are right; below 0.8, only a third to two thirds are. Scoring each allowed reply against the others (choice probability) is less useful: it is over-confident (ECE 0.044), changes the answer on 4 of 2,102 cards, and does not cut wrong saves on real bean speech. Most test cards are easy, so the "all" row is dominated by cards at 0.9 or more; the real Sikia and real bean rows say more about real use. Real bean wrong saves are mostly options added or dropped on multi-choice answers, and a re-ask costs one in five answers there.

Known weak spots:

• Real bean multi-choice answers: options added or dropped (wrong saves 0.254); some of the keys look doubtful and need a review. v8 is level with v7 here (28 cards better, 23 worse).
• Negation on single-choice questions about health ("Wote ni wazima" to "have your animals been sick?") is sometimes read the wrong way round; the training rows covered crop questions only.
• Trailing-off refusals ("Sitaki…") on multi-choice questions were saved as an option once.
• Free text on real Sikia speech (the benchmark keys often summarise).
• Off-topic replies that answer a different question (Sikia's relevance test pending).

Training procedure

LoRA (PEFT) on the language model's linear layers only; the vision and audio towers are untouched. Loss on the JSON reply only. Rank 16, alpha 32, 3 epochs, effective batch 16 (2 × 8), bf16, one GPU, transformers 5.17, seed 42. Settings in train_config.json.