AliceAI-T5-35B-A0.6B · chat + tool-call LoRA, v3

Status: v3, a clear step up but not finished. On the same 72 probes (wording held out from training) v3 passes 88% greedy and 83% sampled, against 63% for v2. Greetings under agent prompts: 24/24 (v2: 11/24). On held-out real turns it picks the right kind of reply 91% of the time, never wrote another turn, made no duplicate calls and at most 4 parallel calls. It still repeats the call it just made instead of answering after a tool result under the pi prompt (after-tool probes: 4/10 greedy, 3/10 sampled; v2: 3/10), and once answered a thank-you with a tool call. Likely cause: the pi prompt never appears in the training data together with a tool result. It missed the pass bar set before the run (90% overall, 90% after-tool, 90% silent), so the 1,625-step run was not done.

A LoRA adapter (rank 16, alpha 16) for yandex/AliceAI-T5-35B-A0.6B, a 35B-total / 0.6B-active sparse-MoE encoder-decoder base model with no chat format. It teaches the model to chat, to call tools when a task needs them, to answer in text when it does not, and to say nothing when there is nothing to say.

How we got here

v1 ran inside an agent harness and misbehaved: a bare "hi" produced 17 to 22 parallel tool calls, "don't read any skills" was ignored, replies were empty or endless. Cause, measured: its chat format used the vocabulary's [INST_START], [REPLY_END], [SYSTEM_*], [TOOL_RESULT_*] tokens, whose embeddings in the base model are untrained placeholders, identical to the unused <SPAN#5000>. The model could not tell roles apart or emit [REPLY_END], and LoRA on attention cannot teach embeddings. placeholder_ids.json lists all 5,832 placeholder tokens, and a unit test fails if the chat format ever uses one.

v2 switched to plain-text role markers and </s> as the end of a turn, and mixed plain chat into the data. It passed 61% of its behaviour probes (base model: 11%) but still, under a coding-agent prompt, invented a task for a bare "hi" and called a tool, repeated a tool call after its result instead of answering, over-thought simple questions, and answered "thanks" with reasoning instead of silence.

v3 changes only the data (same recipe, so any difference is attributable to it): more greetings, small talk and "no tools" instructions under varied agent prompts; real "final answer right after a tool result" turns; reasoning capped at 600 tokens and stripped from a share of targets, so thinking is optional and short; thank-you silence under harness prompts; calls that repeat a recent call are never training targets. The probes were also rebuilt: greeting, negation, small-talk and thank-you wording is split by hash into train and probe pools, so the probes share no wording with the training data (a test asserts it). v2's probes partly overlapped its training phrasings, which flattered v2's negation score.

Chat template

The base repo ships no chat template; chat_template.jinja defines one inside the model's UL2 frame:

encoder:  [_S_]<|system|>…\n<|user|>…\n<|assistant|>…\n<|tool|>…\n<|assistant|><SPAN#0>
decoder:  <s><SPAN#0>  →  <think>reasoning</think>text\n<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call></s>

Tools go in the system message as text in the GLM format (# Tools … <tools>{json per line}</tools> …), produced by glm_tools.py. Reasoning is optional and short. An empty reply is just </s>. Pass alice_frame=False to apply_chat_template when a runtime adds [_S_] and <SPAN#0> itself (vLLM plugin below).

Use

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

hf_hub_download("kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v3", "alice_chat.py", local_dir=".")
from alice_chat import clean_decode, parse_output

BASE, ADAPTER = "yandex/AliceAI-T5-35B-A0.6B", "kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v3"
tok = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)          # carries the chat template
model = AutoModelForSeq2SeqLM.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16,
                                              attn_implementation="eager", device_map={"": "cuda:0"})
model = PeftModel.from_pretrained(model, ADAPTER).eval()     # or load the merged repo directly

messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Give me one fact about Zürich."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
src = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
span = tok.convert_tokens_to_ids("<SPAN#0>")
dec = torch.tensor([[model.config.decoder.bos_token_id, span]], device=model.device)
out = model.generate(input_ids=src, attention_mask=torch.ones_like(src), decoder_input_ids=dec,
                     do_sample=True, temperature=0.6, top_p=0.95, max_new_tokens=512)
gen = out[0, 2:].tolist()
gen = gen[: gen.index(tok.eos_token_id)] if tok.eos_token_id in gen else gen
reasoning, content, tool_calls = parse_output(clean_decode(tok, gen))

Use sampling, not greedy: greedy decoding looped in testing.

Serving with vLLM and an OpenAI-compatible client

The community plugin (AntonProkopyev/AliceAI-T5-35B-A0.6B-vLLM) loads the merged repo unchanged, but it only serves /v1/completions and adds the UL2 frame itself. alice_openai_shim.py sits in front of it and provides /v1/chat/completions with tools, reasoning and streaming (simulated). It also samples by default (the harness sends no temperature), merges duplicate tool calls, caps parallel calls, never lets a turn continue as the user, and retries an empty reply after a user or tool message. serve_smoke.py exercises it with the openai Python client.

Training

Data

One example per assistant turn: the encoder reads everything before it, the decoder learns that turn, then </s>.

Sources: real agent trajectories (UltraData-SFT-Agent-2609, GLM-formatted copy linked above), plain Q&A from UltraData-SFT-2605 (English IF and Knowledge), greetings and system-prompted chats from smoltalk, a sanitized capture of the pi coding-agent prompt and tool schemas (harness_pi.json), and templated greetings, small talk, "no tools" instructions and thank-yous (phrases.py). Targets are filtered to at most 8 parallel calls, 600 reasoning tokens and no repeat of a recent call. Held-out rows are never trained on.

Evaluation

Behaviour probes (probes.py): greetings, "don't use tools", small talk and plain questions under pi-style prompts (with and without a 14-skill list, and with a tool subset) and a non-coding weather / calculator / search assistant; real actions; answering after a tool result; silence after a thank-you. A probe passes only if every expectation holds, the reply finished, and it never wrote another turn. The previous adapter is scored on the same probes.

Held-out real turns (this adapter):

n = 60 held-out rows, greedy, 1,024 new tokens.

These are small samples (72 probes, a few dozen held-out rows) of greedy or single-sample generations. They check for the specific failures above; they are not a capability benchmark. Tool-call arguments are not checked, only the tool name, and the tool-name metric counts a reasonable alternative as wrong.

Limitations

• Small budget. 200 steps is 1,600 examples: enough to move behaviour, not to add agentic competence the base lacks.
• Attention-only adaptation. Experts and embeddings are untouched.
• Templated data. Greetings, small talk, negations and thank-yous are templated, which is why a hash-held-out 20% of their wording is kept for the probes; real-world phrasing outside those templates is untested.
• Tokenizer quirk. The tokenizer inserts a space after every newline when decoding; use clean_decode from alice_chat.py (important for code in tool arguments).
• Data. Reasoning traces come from whatever model produced the upstream data, not from AliceAI. Plain chat comes from English-only sources. No safety tuning was done: the model can and will run bash commands if a harness lets it.
• vLLM. The merged-weights path was verified with v1's checkpoint on the community plugin; re-verify for this one.

Licence and attribution

Apache-2.0, matching the base model (© Yandex LLC) and the data (UltraData: arXiv:2602.09003; smoltalk: Apache-2.0 for its new subsets).