MiniCPM-V-4.7-1B-Agentic-FwS-FlashREINFORCE
ewin-reg/MiniCPM-V-4.7-1B-Agentic-FwS (v6) after 40 FlashREINFORCE updates on fresh, grounded Nemotron tool-calling prompts.
Statistically a tie with v6, not a proven improvement. These are the same weights as main of the main repo (v9), which the owner chose to make main; v6 stays at revision f7c6ea0 (tag v6-round2). This repo keeps the standalone card for the experiment.
Method
FlashREINFORCE (NVIDIA, alphaXiv 2609.flashreinforce-asynchronous-rl-agentic-models): one rollout per prompt, binary reward (the first tool call's text equals the gold call, the same "exact" as the val metric), advantage = reward minus the batch mean, one optimizer step per fresh batch, sequence-level trust region on the mean per-token Bernoulli KL proxy (threshold 1e-3), sample-mean loss, no critic, no reference model, no KL penalty.
- Start: v6 (
ar2_sft7), LoRA r32 alpha 64, merged into round 1 (revision ab2a7d9) for this release. - Data: Nemotron-SFT-Agentic-v4
tool_callingrows 70000 and later (v8 stopped at row 70000), only examples whose gold argument values all appear in the prompt. Prompts equal to any earlier training, val or bench prompt were removed. 40 updates x 64 prompts = 2,560 prompts, each used once. - Sampling: temperature 1.0, top_p 1.0, top_k 0, up to 768 new tokens.
- Non-default values: learning rate 1e-5 (the paper's 1e-6 is for full fine-tuning; no LoRA value was measured), LoRA dropout 0, batch 64 (paper 128; T4 time), constant learning rate, fp16 with GradScaler.
Results (1000 held-out val rows, greedy, first tool call)
| exact | args | name | has_call | |
|---|---|---|---|---|
| round 1 | 0.314 | 0.337 | 0.754 | 0.997 |
v6 (main repo, revision f7c6ea0) |
0.409 | 0.440 | 0.832 | 0.999 |
| this model | 0.414 | 0.440 | 0.831 | 1.000 |
| reference: Claude Haiku 5.5, run as a subagent (not a MiniCPM model) | 0.334 | 0.372 | 0.724 | 0.895 |
Against v6: exact +0.005, paired SE 0.0102 (55 wins, 50 losses), below the 2 SE keep threshold; name is 0.001 lower. On the 584 val rows whose gold values all appear in the prompt: 0.647 against 0.627 for v6, within noise. Against Claude Haiku 5.5 on the same 1000 rows (paired): this model is ahead by exact +0.080 (SE 0.0141), args +0.068 (SE 0.0147), name +0.107 (SE 0.0161). Haiku was run as a Claude Code subagent, 25 prompts per agent, under the agent's own system prompt; 105 of its 1000 replies were plain text although every gold answer is a tool call, and on chained requests it often sent only the first call. The rows are in Nemotron's own style, which this model trained on, so this shows agreement with the dataset, not general tool-use ability. Some batches were rerun after the agents lost context; a few kept rows were flagged as inferred and were not re-verified. During training the drift check rejected 0 of 64 trajectories at every update (proxy about 4e-6), and the share of outputs with a tool call stayed at 1.0. Greedy exact on 64 held-out fresh rows: 0.609 at the start, 0.594 at update 40.
Limitations
- No measurable gain over v6. 40 updates at LoRA lr 1e-5 may be too few; this was not tested.
- Same limits as the main repo: the metric needs the whole call text to match, and about 60% of gold calls with checkable values contain values that are not in the prompt.