AliceAI-T5-35B-A0.6B · chat + tool-call LoRA, v4

Status: v4, fixes the failure it targeted but is not yet robust. After a tool result under pi's prompt it now answers instead of repeating the call (after-tool probes 22/23; v3: 8/23 on the same probes). It passes 90% of the 103 probes greedy (v3: 80%) but only 80% sampled at T=0.6, a gap that suggests its decisions are not yet confident. Regressions against v3: a plain general question under pi's prompt sometimes triggers a pointless file read (2/4 greedy), "don't use tools" is ignored in 3 of 8 negation probes, and on held-out real turns the right-kind-of-reply rate fell from 91% to 83% (7 of 38 tool-call turns were answered in text instead). It missed the bar set before the run (sampled probes ≥ 90%, held-out decision accuracy ≥ 88%, no turn-marker leaks), so it was not scaled up.

A LoRA adapter (rank 16, alpha 16) for yandex/AliceAI-T5-35B-A0.6B, a 35B-total / 0.6B-active sparse-MoE encoder-decoder base model with no chat format. It teaches the model to chat, to call tools when a task needs them, to answer in text when it does not, and to say nothing when there is nothing to say.

How we got here

v1 ran inside an agent harness and misbehaved: a bare "hi" produced 17 to 22 parallel tool calls, "don't read any skills" was ignored, replies were empty or endless. Cause, measured: its chat format used the vocabulary's [INST_START], [REPLY_END], [SYSTEM_*], [TOOL_RESULT_*] tokens, whose embeddings in the base model are untrained placeholders, identical to the unused <SPAN#5000>. The model could not tell roles apart or emit [REPLY_END], and LoRA on attention cannot teach embeddings. placeholder_ids.json lists all 5,832 placeholder tokens, and a unit test fails if the chat format ever uses one.

v2 switched to plain-text role markers and </s> as the end of a turn, and mixed plain chat into the data. It passed 61% of its behaviour probes (base model: 11%) but still, under a coding-agent prompt, invented a task for a bare "hi" and called a tool, repeated a tool call after its result instead of answering, over-thought simple questions, and answered "thanks" with reasoning instead of silence.

v3 changed only the data: more greetings, small talk and "no tools" instructions under varied agent prompts, real "final answer right after a tool result" turns, reasoning capped at 600 tokens and optional, thank-you silence under harness prompts, and no training target that repeats a recent call. It passed 88% of the (rebuilt) probes, with greetings at 24/24, but under pi's prompt it still repeated the call it had just made instead of answering once a tool result arrived (after-tool probes 4/10). The cause was visible in the data: pi's prompt only ever appeared with plain replies, never next to a tool result.

v4 changes only the data again, to put exactly that into training:

• Retargeted real trajectories. General-Agent's tools (read, exec, edit, write) are nearly pi's (read, bash, edit, write). About 85% of its calls map, but no whole trajectory does (each ends up calling something unmappable), so only the part of a trajectory before its first unmappable call is used: tool names mapped, {"result": ...} results unwrapped, edit arguments parsed, and pi's system prompt and tool schemas swapped in.
• Templated pi flows (pi_flows.py): read a value, a list, a note or code; ls; cat; wc -l; write; a two-step edit (read, then an edit that differs from the read); compare two files. Results and final answers are deterministic.
• Thank-you silence after a finished pi task.
• The probes were rebuilt again: 103 of them, with slot values (files, contents, numbers) and phrasings that the flow generator never emits for training, a probe type for the second call of a multi-step flow (it must differ from the first), and a scorer check for repeated calls. (Writing that check exposed a bug in my first scorer, which could never have flagged a repeat.)

Chat template

The base repo ships no chat template; chat_template.jinja defines one inside the model's UL2 frame:

encoder:  [_S_]<|system|>…\n<|user|>…\n<|assistant|>…\n<|tool|>…\n<|assistant|><SPAN#0>
decoder:  <s><SPAN#0>  →  <think>reasoning</think>text\n<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call></s>

Tools go in the system message as text in the GLM format (# Tools … <tools>{json per line}</tools> …), produced by glm_tools.py. Reasoning is optional and short. An empty reply is just </s>. Pass alice_frame=False to apply_chat_template when a runtime adds [_S_] and <SPAN#0> itself (vLLM plugin below).

Use

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

hf_hub_download("kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v4", "alice_chat.py", local_dir=".")
from alice_chat import clean_decode, parse_output

BASE, ADAPTER = "yandex/AliceAI-T5-35B-A0.6B", "kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v4"
tok = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)          # carries the chat template
model = AutoModelForSeq2SeqLM.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16,
                                              attn_implementation="eager", device_map={"": "cuda:0"})
model = PeftModel.from_pretrained(model, ADAPTER).eval()     # or load the merged repo directly

messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Give me one fact about Zürich."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
src = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
span = tok.convert_tokens_to_ids("<SPAN#0>")
dec = torch.tensor([[model.config.decoder.bos_token_id, span]], device=model.device)
out = model.generate(input_ids=src, attention_mask=torch.ones_like(src), decoder_input_ids=dec,
                     do_sample=True, temperature=0.6, top_p=0.95, max_new_tokens=512)
gen = out[0, 2:].tolist()
gen = gen[: gen.index(tok.eos_token_id)] if tok.eos_token_id in gen else gen
reasoning, content, tool_calls = parse_output(clean_decode(tok, gen))

Use sampling, not greedy: greedy decoding looped in testing.

Serving with vLLM and an OpenAI-compatible client

The community plugin (AntonProkopyev/AliceAI-T5-35B-A0.6B-vLLM) loads the merged repo unchanged, but it only serves /v1/completions and adds the UL2 frame itself. alice_openai_shim.py sits in front of it and provides /v1/chat/completions with tools, reasoning and streaming (simulated). It also samples by default (the harness sends no temperature), merges duplicate tool calls, caps parallel calls, never lets a turn continue as the user, and retries an empty reply after a user or tool message. serve_smoke.py exercises it with the openai Python client.

Training

Data

One example per assistant turn: the encoder reads everything before it, the decoder learns that turn, then </s>.

Sources: real agent trajectories (UltraData-SFT-Agent-2609, GLM-formatted copy linked above), plain Q&A from UltraData-SFT-2605 (English IF and Knowledge), greetings and system-prompted chats from smoltalk, a sanitized capture of the pi coding-agent prompt and tool schemas (harness_pi.json), General-Agent trajectories retargeted onto pi's tools, templated pi flows (pi_flows.py), and templated greetings, small talk, "no tools" instructions and thank-yous (phrases.py). Targets are filtered to at most 8 parallel calls, 600 reasoning tokens and no repeat of a recent call. Held-out rows are never trained on.

Evaluation

Behaviour probes (probes.py): greetings, "don't use tools", small talk and plain questions under pi-style prompts (with and without a 14-skill list, and with a tool subset) and a non-coding weather / calculator / search assistant; real first calls; answering after a tool result; the second call of a multi-step task; silence after a thank-you or a finished task. A probe passes only if every expectation holds, the reply finished, and it never wrote another turn. The previous adapter (v3) is scored on the same probes.

Held-out real turns (this adapter):

n = 60 held-out rows, greedy, 1,024 new tokens.

These are small samples (103 probes, a few dozen held-out rows) of greedy or single-sample generations. They check for the specific failures above; they are not a capability benchmark. Tool-call arguments are not checked, only the tool name, and the tool-name metric counts a reasonable alternative as wrong.

Limitations

• Small budget. 200 steps is 1,600 examples: enough to move behaviour, not to add agentic competence the base lacks.
• Attention-only adaptation. Experts and embeddings are untouched.
• Templated data. Greetings, small talk, negations, thank-yous and the pi flows are templated, which is why a hash-held-out 20% of their wording is kept for the probes; real-world phrasing outside those templates is untested.
• Tokenizer quirk. The tokenizer inserts a space after every newline when decoding; use clean_decode from alice_chat.py (important for code in tool arguments).
• Data. Reasoning traces come from whatever model produced the upstream data, not from AliceAI. Plain chat comes from English-only sources. No safety tuning was done: the model can and will run bash commands if a harness lets it.
• vLLM. The merged-weights path was verified with v1's checkpoint on the community plugin; re-verify for this one.

Licence and attribution

Apache-2.0, matching the base model (© Yandex LLC) and the data (UltraData: arXiv:2602.09003; smoltalk: Apache-2.0 for its new subsets).