AliceAI-T5-35B-A0.6B · chat + tool-call LoRA

Status: experimental v1, known issues (updated 2026-10-01). Not recommended as an agent. Tested in an agent harness, this adapter over-uses tools (a bare "hi" produced 17 to 22 parallel tool calls), ignores "do not use tools" instructions, sometimes answers with nothing, can run on without stopping, and ends turns with [COT_END] instead of [REPLY_END].

Measured cause. The delimiter tokens this chat format is built on ([INST_START], [REPLY_END], [SYSTEM_*], [TOOL_RESULT_*], ...) are untrained placeholders in the base model: their embeddings are identical to the never-used <SPAN#5000> ([REPLY_END], [TOOL_RESULT_*] bit-identical, the rest within 0.004). They are therefore indistinguishable to the model, and LoRA on attention cannot teach them, so the model cannot tell roles apart or emit [REPLY_END]. Secondary cause: outside the Tool-Use subsplit about 98% of training turns are tool calls. A v2 without placeholder tokens is in progress. The evaluation below predates this finding and only measures format.

A LoRA adapter (rank 16, alpha 16) for yandex/AliceAI-T5-35B-A0.6B, a 35B-total / 0.6B-active sparse-MoE encoder-decoder base model. The base model is a span-infilling (UL2) pretrain with no chat format. This adapter teaches it three things:

1. Chat: multi-turn conversation in a fixed template.
2. Tool calls: emit <tool_call>…</tool_call> when a tool is needed, read [TOOL_RESULT_*] turns, answer.
3. When to stop: end every reply with [REPLY_END]</s>, never write the user's next turn, and produce nothing (just </s>) when the conversation already ends on a finished assistant reply.

Chat template

The base repo ships no chat_template.jinja; chat_template.jinja here defines one. It uses delimiter tokens that already exist in the vocabulary (no resize) inside the model's UL2 frame:

encoder:  [_S_][SYSTEM_START]…[SYSTEM_END][INST_START]…[INST_END][REPLY_START]…[REPLY_END][TOOL_RESULT_START]…[TOOL_RESULT_END]…[REPLY_START]<SPAN#0>
decoder:  <s><SPAN#0>  →  [COT_START]reasoning[COT_END] text\n<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>[REPLY_END]</s>

Tools go in the system message as text, in the format GLM-5.3-Flash's own template renders (# Tools … <tools>{json per line}</tools> … <tool_call>{function-name}<arg_key>…). The training data already has this block; see the dataset card. Reasoning is rendered only for the turn being predicted and dropped from history.

Use

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

hf_hub_download("kaivoss/AliceAI-T5-35B-A0.6B-chat-lora", "alice_chat.py", local_dir=".")
from alice_chat import clean_decode, parse_tool_calls

BASE, ADAPTER = "yandex/AliceAI-T5-35B-A0.6B", "kaivoss/AliceAI-T5-35B-A0.6B-chat-lora"
tok = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)          # carries the chat template
model = AutoModelForSeq2SeqLM.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16,
                                              attn_implementation="eager", device_map={"": "cuda:0"})
model = PeftModel.from_pretrained(model, ADAPTER).eval()     # or load the merged repo directly

messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Give me one fact about Zürich."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
src = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
span = tok.convert_tokens_to_ids("<SPAN#0>")
dec = torch.tensor([[model.config.decoder.bos_token_id, span]], device=model.device)
with torch.inference_mode():
    out = model.generate(input_ids=src, attention_mask=torch.ones_like(src), decoder_input_ids=dec,
                         do_sample=False, max_new_tokens=512)
gen = out[0, 2:].tolist()
gen = gen[: gen.index(tok.eos_token_id)] if tok.eos_token_id in gen else gen
text = clean_decode(tok, gen)            # see "Tokenizer quirk" below; empty string = "nothing to say"
print(text, parse_tool_calls(text))

Fine-tuning example

finetune_chat_lora.py is the script that produced this adapter. It is the model repo's finetune_example.py with pre-tokenised 32k-context examples, r=16/α=16, an attention-backend fallback list and peak-memory logging. prepare_data.py builds the examples, eval_chat.py the base-vs-LoRA comparison, run.sh the exact pod pipeline.

pip install -r requirements.txt
python prepare_data.py --out data                                   # one example per assistant turn
python finetune_chat_lora.py --data data/train.jsonl --out adapter  # 200 steps, one 141 GB GPU

Each example keeps the repo's layout: input_ids=[[_S_] … <SPAN#0>], decoder_input_ids=[<s>, <SPAN#0>, *target], labels=[-100, *target, </s>]. The loss is on the target only, and the target always ends in </s>.

Training

The experts are fused nn.Parameters rather than nn.Linear, so LoRA cannot wrap them: only attention is adapted.

Data

One example per assistant turn: the encoder reads everything before it, the decoder learns that turn. About 10% are "stay silent" examples whose source already ends on a finished assistant reply and whose target is just </s>. Equal numbers come from each upstream subsplit.

Evaluation

Held-out examples from the same distribution, greedy decoding, base model and adapter run through the same prompts.

n = 40 held-out examples, greedy decoding.

Read these numbers with care. The eval is small (n = 40, only 2 non-silent Tool-Use examples) and generation is capped at 384 new tokens. Code, General and Search turns usually open with a long [COT_START] reasoning block, so a capped output is indistinguishable from a runaway one: clean stop and tool-call recall are lower bounds, not stop/recall rates. What the per-row data does support:

• The clear effect is the silent behaviour: the base model answered a finished conversation with another tool call on 5 of 6 rows; the adapter emits </s>.
• Tool-call quality did not improve. Overall recall is identical (22/32), tool-name agreement went down, and Search-Agent recall fell from 10/11 to 5/11. The 384-token cap probably explains part of that (the call never got generated), but this run cannot separate truncation from a real regression.
• Neither model wrote a user turn in this sample, so nothing here shows the adapter fixing that failure mode.

Training loss (mean of 8-example steps, so noisy): about 1.3 at the start, mean 0.84 over the 200 steps, 0.5-1.1 by the end. The raw per-example outputs are in eval_results.json; a proper eval needs a larger sample, a larger token cap, and execution of the tool calls.

Limitations

• Small budget. 200 steps is 1,600 examples. That is enough to teach a format and stopping behaviour; it is not enough to add agentic competence the base model lacks. Expect format learned, reasoning quality mostly the base model's.
• Attention-only adaptation (see above).
• Tokenizer quirk. The tokenizer is Metaspace(prepend_scheme="always") with \n as its own token, so tok.decode inserts a space after every newline and delimiter. Use clean_decode from alice_chat.py, especially for code in tool arguments. A line with exactly one leading space is indistinguishable from one with none at the token level.
• Attention backend. The remote code does not declare SDPA support, so finetune_chat_lora.py sets _supports_sdpa itself (checked numerically against eager, including padded batches, on a tiny model). The inference snippet above uses eager; at 32k prompts use the same SDPA patch or flash-attn.
• Data. Rows come from the head of each upstream shard, not a random sample; reasoning traces come from whatever model produced the upstream trajectories, not from AliceAI. No safety tuning was done.
• Decoding. Only greedy decoding was evaluated.

Licence and attribution

Apache-2.0, matching the base model (yandex/AliceAI-T5-35B-A0.6B, © Yandex LLC) and the data (openbmb/UltraData-SFT-Agent-2609, arXiv:2602.09003).