AliceAI-T5-35B-A0.6B · chat + tool-call LoRA, v5

Status: v5, the most robust version so far, still short of the bar. On 129 probes (all wording held out) v5 passes 88% greedy and 85% sampled (v4: 81% greedy on the same probes). Greetings 24/24, plain questions under pi 8/8, "don't use tools" 8/8 and silence 15/15, in both modes, and sampled is now within 3 points of greedy (v4: 10). On held-out real turns it picks the right kind of reply 96% of the time (v4: 83%, v3: 91%), never wrote another turn and made no duplicate calls. Still failing: a task that needs a tool plus "don't use tools" (2/8: it usually calls a tool anyway), some answers after a tool result (25/31: it sometimes puts the answer inside <think>, leaves out the key fact, or calls again), and in one probe it printed file contents it had never read. It missed the bar set before the run (probes ≥ 90%, after-tool ≥ 90%), so it was not scaled up.

A LoRA adapter (rank 16, alpha 16) for yandex/AliceAI-T5-35B-A0.6B, a 35B-total / 0.6B-active sparse-MoE encoder-decoder base model with no chat format. It teaches the model to chat, to call tools when a task needs them, to answer in text when it does not, and to say nothing when there is nothing to say.

How we got here

v1 ran inside an agent harness and misbehaved: a bare "hi" produced 17 to 22 parallel tool calls, "don't read any skills" was ignored, replies were empty or endless. Cause, measured: its chat format used the vocabulary's [INST_START], [REPLY_END], [SYSTEM_*], [TOOL_RESULT_*] tokens, whose embeddings in the base model are untrained placeholders, identical to the unused <SPAN#5000>. The model could not tell roles apart or emit [REPLY_END], and LoRA on attention cannot teach embeddings. placeholder_ids.json lists all 5,832 placeholder tokens, and a unit test fails if the chat format ever uses one.

v2 switched to plain-text role markers and </s> as the end of a turn, and mixed plain chat into the data. It passed 61% of its behaviour probes (base model: 11%) but still, under a coding-agent prompt, invented a task for a bare "hi" and called a tool, repeated a tool call after its result instead of answering, over-thought simple questions, and answered "thanks" with reasoning instead of silence.

v3 changed only the data: more greetings, small talk and "no tools" instructions under varied agent prompts, real "final answer right after a tool result" turns, reasoning capped at 600 tokens and optional, thank-you silence under harness prompts, and no training target that repeats a recent call. It passed 88% of the (rebuilt) probes, with greetings at 24/24, but under pi's prompt it still repeated the call it had just made instead of answering once a tool result arrived (after-tool probes 4/10). The cause was visible in the data: pi's prompt only ever appeared with plain replies, never next to a tool result.

v4 changed only the data again, to put exactly that into training: real General-Agent trajectory prefixes retargeted onto pi's tools and prompt (about 85% of its calls map onto read/bash/edit/write, but no whole trajectory does, so only the part before the first unmappable call is used), plus templated pi flows with deterministic results and answers. It fixed the bug: after-tool probes went from 8/23 (v3) to 22/23. But it regressed elsewhere: a plain question under pi's prompt got a pointless file read (2/4 greedy), "don't use tools" was ignored in 3 of 8 probes, and on held-out real turns the right-kind-of-reply rate fell from 91% to 83% (it under-called tools). Its sampled score was 10 points below its greedy score, so many decisions were close calls.

v5 changes only the data once more, aimed at those regressions:

• pi_plain: real plain questions under pi and agent prompts with no "no tools" instruction. In v4 the only plain real questions under pi's prompt carried one, so pi's prompt had come to mean "call a tool".
• negation: "don't use tools" under agent prompts, both greeting plus instruction and a task that would need a tool plus the instruction (the right reply is an honest text answer: "I can't see config.yaml without reading it; paste it"), and neither a guess nor a call.
• Rebalanced towards calling tools (more real tool-call turns, fewer text turns) to fix the under-calling on held-out turns.
• Richer flows: four new task types (grep -c, head, mkdir with an empty result, find), larger pools (about 1,400 distinct flows per 2,000 draws, up from about 770) and an even split between first calls and final answers.
• More silence after a thank-you, including after a finished pi task.
• 129 probes (up from 103), adding plain questions under a long-skills pi prompt and negation-with-task probes. Every slot value, phrasing and instruction in the probes is held out of training by construction (a test asserts it).

Chat template

The base repo ships no chat template; chat_template.jinja defines one inside the model's UL2 frame:

encoder:  [_S_]<|system|>…\n<|user|>…\n<|assistant|>…\n<|tool|>…\n<|assistant|><SPAN#0>
decoder:  <s><SPAN#0>  →  <think>reasoning</think>text\n<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call></s>

Tools go in the system message as text in the GLM format (# Tools … <tools>{json per line}</tools> …), produced by glm_tools.py. Reasoning is optional and short. An empty reply is just </s>. Pass alice_frame=False to apply_chat_template when a runtime adds [_S_] and <SPAN#0> itself (vLLM plugin below).

Use

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

hf_hub_download("kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v5", "alice_chat.py", local_dir=".")
from alice_chat import clean_decode, parse_output

BASE, ADAPTER = "yandex/AliceAI-T5-35B-A0.6B", "kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v5"
tok = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)          # carries the chat template
model = AutoModelForSeq2SeqLM.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16,
                                              attn_implementation="eager", device_map={"": "cuda:0"})
model = PeftModel.from_pretrained(model, ADAPTER).eval()     # or load the merged repo directly

messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Give me one fact about Zürich."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
src = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
span = tok.convert_tokens_to_ids("<SPAN#0>")
dec = torch.tensor([[model.config.decoder.bos_token_id, span]], device=model.device)
out = model.generate(input_ids=src, attention_mask=torch.ones_like(src), decoder_input_ids=dec,
                     do_sample=True, temperature=0.6, top_p=0.95, max_new_tokens=512)
gen = out[0, 2:].tolist()
gen = gen[: gen.index(tok.eos_token_id)] if tok.eos_token_id in gen else gen
reasoning, content, tool_calls = parse_output(clean_decode(tok, gen))

Use sampling, not greedy: greedy decoding looped in testing.

Serving with vLLM and an OpenAI-compatible client

The community plugin (AntonProkopyev/AliceAI-T5-35B-A0.6B-vLLM) loads the merged repo unchanged, but it only serves /v1/completions and adds the UL2 frame itself. alice_openai_shim.py sits in front of it and provides /v1/chat/completions with tools, reasoning and streaming (simulated). It also samples by default (the harness sends no temperature), merges duplicate tool calls, caps parallel calls, never lets a turn continue as the user, retries an empty reply after a user or tool message, shows an answer that landed inside <think> as the reply, and cuts a repetition loop after its first cycle (v5 once looped a story for 13,221 tokens). serve_smoke.py exercises it with the openai Python client.

Training

Data

One example per assistant turn: the encoder reads everything before it, the decoder learns that turn, then </s>.

Sources: real agent trajectories (UltraData-SFT-Agent-2609, GLM-formatted copy linked above), plain Q&A from UltraData-SFT-2605 (English IF and Knowledge), greetings and system-prompted chats from smoltalk, a sanitized capture of the pi coding-agent prompt and tool schemas (harness_pi.json), General-Agent trajectories retargeted onto pi's tools, templated pi flows and negation-with-task replies (pi_flows.py), and templated greetings, small talk, "no tools" instructions and thank-yous (phrases.py). Targets are filtered to at most 8 parallel calls, 600 reasoning tokens and no repeat of a recent call. Held-out rows are never trained on.

Evaluation

Behaviour probes (probes.py): greetings, "don't use tools", small talk and plain questions under pi-style prompts (with and without a 14-skill list, and with a tool subset) and a non-coding weather / calculator / search assistant; real first calls; answering after a tool result; the second call of a multi-step task; silence after a thank-you or a finished task. A probe passes only if every expectation holds, the reply finished, and it never wrote another turn. The previous adapter (v3) is scored on the same probes.

Held-out real turns (this adapter):

n = 60 held-out rows, greedy, 1,024 new tokens.

These are small samples (129 probes, a few dozen held-out rows) of greedy or single-sample generations. They check for the specific failures above; they are not a capability benchmark. Tool-call arguments are not checked, only the tool name, and the tool-name metric counts a reasonable alternative as wrong.

Limitations

• Small budget. 200 steps is 1,600 examples: enough to move behaviour, not to add agentic competence the base lacks.
• Attention-only adaptation. Experts and embeddings are untouched.
• Templated data. Greetings, small talk, negations, thank-yous and the pi flows are templated, which is why a hash-held-out 20% of their wording is kept for the probes; real-world phrasing outside those templates is untested.
• Tokenizer quirk. The tokenizer inserts a space after every newline when decoding; use clean_decode from alice_chat.py (important for code in tool arguments).
• Data. Reasoning traces come from whatever model produced the upstream data, not from AliceAI. Plain chat comes from English-only sources. No safety tuning was done: the model can and will run bash commands if a harness lets it.
• vLLM. The merged-weights path was verified with v1's checkpoint on the community plugin; re-verify for this one.

Licence and attribution

Apache-2.0, matching the base model (© Yandex LLC) and the data (UltraData: arXiv:2602.09003; smoltalk: Apache-2.0 for its new subsets).