Qwen3.5-0.8B-ConvFill

A full fine-tune of Qwen/Qwen3.5-0.8B on the ConvFill task from Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents (arXiv 2511.07397). It plays the paper's Talker role: a small on-device model that speaks one conversational phrase per streamed item from a cloud Reasoner — a knowledge chunk, or a <sil> token when no chunk has arrived in the last second.

Task format

One training example is one phrase. The prompt contains (in order) the previous turn's history, the current user utterance, the phrases already spoken this turn, and the current stream item:

<|im_start|>user
{previous user turn}<|im_end|>
<|im_start|>assistant
{previous responder turn}<|im_end|>
<|im_start|>user
{current user utterance}<|im_end|>
<|im_start|>assistant
{previous phrases this turn}<|im_end|>
<|im_start|>knowledge
{knowledge chunk | <sil>}<|im_end|>
<|im_start|>assistant
{current phrase}<|im_end|>

• Loss is computed only on the final response span (current phrase + <|im_end|>\n); all prefix labels are masked to -100.
• <sil> is added as a special token (id 248077; the model's 248,320-row embedding already covered it, so no resize was needed).
• Uses the Qwen3.5 <|im_start|>/<|im_end|> markers, same as the paper's Qwen3-0.6B Talker (vysri/Qwen-0.6B-ConvFill); Qwen3.5 is loaded through its text-only path (Qwen3_5ForCausalLM).

Training

• Data: zenglhardt/convfill-dataset (CC BY-SA 4.0) — all 7 domain files, preprocessed with conversation history included (the paper's dataset_preprocess.py --include_history port). 261,354 train / 29,217 val phrase examples from an 8,443-conversation 90/10 split by conversation (seed 42); ~50M train tokens, mean length 192, max 487.
• Recipe (paper): full fine-tune, AdamW (fused), lr 1e-4, weight decay 0.01, cosine schedule, 200 warmup steps, effective batch 16 (per-device 2 × accumulation 8), bf16, 1 epoch = 16,335 optimizer steps.
• Hardware: Hugging Face Jobs on a10g-small, ~1.45 s/optimizer-step.
• Result: final validation loss 0.0370 (150 held-out turns, ~1,570 phrase examples; step-2500 eval earlier in training was 0.0048 on a different val subset draw). Training loss ended in the ~1e-4–1e-3 band.

Honest notes / deviations

• The run was executed across resumed legs. Training containers were repeatedly evicted by the infrastructure (~every 1–3 h, no in-process error), so the run was continued from Hub-pushed checkpoints (training_state.json marker + commit-titled checkpoints at effective steps 4000, 5500, 7000, 8500, 10000, 11500, 13000, 14500, 16000). Weights-only resume: the optimizer state and LR schedule restarted at each resume (warmup + cosine over the remaining steps), so the schedule shape differs from the paper's single continuous run. All 16,335 steps of the 1-epoch recipe were completed.
• The paper's headline metric was not reproduced here. The paper reports streaming end-to-end QA accuracy on 250 NaturalQuestions queries with a large cloud Reasoner backend (e.g. 10% → 46–52% for a 360M Talker) plus latency and NLI entailment scores. This checkpoint has not been evaluated that way — the number above is validation next-phrase loss only. To benchmark it properly, run the streaming pipeline from vysri/conversational-infill with this model as the Talker.
• Base model is a multimodal VLM; this fine-tune trains and runs its text path only and does not preserve image/video capabilities.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("andito/Qwen3.5-0.8B-ConvFill")
model = AutoModelForCausalLM.from_pretrained("andito/Qwen3.5-0.8B-ConvFill")

prompt = (
    "<|im_start|>user\nWhat is the treatment for a mild fever?<|im_end|>\n"
    "<|im_start|>knowledge\nRest, fluids, and acetaminophen as needed.<|im_end|>\n"
    "<|im_start|>assistant\n"
)
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

In the real system the Talker generates one phrase per stream item, and a <sil> in the knowledge slot cues ungrounded filler speech.

Citation

@article{convfill2025,
  title={Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents},
  author={Srinivas and Englhardt and Powers and Patel and Iyer},
  journal={arXiv preprint arXiv:2511.07397},
  year={2025},
  url={https://arxiv.org/abs/2511.07397}
}