Introducing ⚡ GLM-4.7-Flash-Coder ⚡

We fine-tuned GLM-4.7-Flash to push its agentic coding capabilities further: from 33.15% to 41.65% on SWE-rebench-V2 (+26% relative improvement) at 14% lower inference cost.

This 30B Mixture-of-Experts model beats competitors 3.5-4× its size — Qwen3.5-122B-A10B and Devstral-2 — and surpasses Ling-2.6-Flash by a narrow margin.

We trained it with Halo, the post-training framework we’re open-sourcing today — the full story of the framework here.

More reasoning, fewer turns

Although GLM-4.7-Flash-Coder does 2.4× more reasoning than the base model, it completes tasks faster: 45% fewer turns, which translates to 14% lower inference cost.

The key to turn efficiency is concurrent tool calls

The fine-tuned model learned to batch its tool use instead of working one call at a time. Turns with two or more parallel read calls jumped from 1.6% to 9.7% of all turns, parallel bash from 0.6% to 7.2%, and parallel edit, which is almost nonexistent in the base model, grew ~90×. Reading three files in one turn instead of three turns is exactly the kind of habit that compounds into 45% fewer turns overall.

SWE-rebench-V2

We trained and evaluated models on nebius/SWE-rebench-V2, a dataset of 32k agentic coding tasks. It consists of real issues from public repositories, closely representing real-world SWE challenges.

Teacher — GLM-5.1

The training data comes from GLM-5.1 acting as a teacher. We ran it on the train split with 4 rollouts per task and kept only trajectories that passed the tests: 7,777 successful multi-turn traces after rejection sampling.

We’re releasing the training dataset alongside the train/test splits.

All models run inside PI — a lightweight agent harness with four tools: bash, read, edit, write.

Training

We trained Coder with Supervised Fine-Tuning (SFT) using Halo, our post-training framework we’re open-sourcing today 🎉

• Parallelism mode: DP=16 (multi-node training)
• Context length: 75k
• Packing: Enabled
• Global batch: 16
• Learning rate scheduling: Cosine, with linear warm-up
• Training time: 3 hours on 16 B300 GPUs

Halo 😇 is driven entirely by a single YAML config

model_name_or_path: zai-org/GLM-4.7-Flash
trust_remote_code: true
attn_implementation: flash_attention_4_hybrid

dataset:
- whitecircle/swe-rebench-v2-glm-5.1-pi-agent-successful-traces
test_size: 0.01

conversation_field: messages
tools_field: tools
interleaved_thinking: true

max_length: 75000
packing: true

learning_rate: 1e-5
lr_scheduler_type: cosine
warmup_steps: 25
per_device_train_batch_size: 1
per_device_eval_batch_size: 1

moe_balancing: none
liger_kernel_config:
  fused_linear_cross_entropy: true

assistant_message_template: "<|assistant|>"

output_dir: /mnt/research/GLM-4.7-Flash-Coder
save_steps: 50
save_only_model: true

eval_strategy: steps
eval_steps: 50
logging_steps: 1
logging_first_step: true
log_decoded_samples: true

report_to: wandb
project_name: agentic-sft
run_name: glm-4.7-flash-coder

enable_efficiency_metrics: true

Emergent skills

The model picked up several of the teacher’s skills and patterns during fine-tuning:

• A 10× higher rate of self-corrections in reasoning,
• Self-directed questions (“But wait, could this ever be True?”),
• Advanced tool calling patterns, e.g. setting timeout for the bash tool.

Perhaps the most surprising transfer was the git stash usage:

Hmm, actually these look like pre-existing test differences unrelated to my fix. Let me revert my fix and run the test again.

git stash && python -m pytest tests/test_markdown.py::test_markdown_table -xvs 2>&1 | tail -20

Loss decomposition into Reasoning vs. Tool Calls

We only train on assistant responses. Of the 177 million tokens in the dataset, only 46 million carry loss — the remaining 131 million (system prompts and tool responses) are masked.

Within an assistant message there are two very different kinds of tokens, mainly reasoning and tool calls. Tool-call tokens are far lower-entropy:

Although tool-call tokens make up 43.3% of input tokens, they contribute only 16.1% of the overall loss. To track convergence separately, we decompose the loss into two components: negative log-likelihood on reasoning tokens and on tool-call tokens.

The training-loss curves show 10–20% cliffs at epoch boundaries — the usual signature of memorizing repeated data. We read it as benign: eval losses keep improving through the second epoch.

It’s possible to set different learning rates for the two components, but on our data both eval losses reach their global minima at the same moment (the end of epoch 2) so we didn’t need to.

Test scores

Below is the comparison of GLM-4.7-Flash, GLM-4.7-Flash-Coder, and its teacher GLM-5.1 on test set of SWE-rebench-V2:

How we enabled FA4

FlashAttention-4 is a fast attention kernel — but it’s incompatible with GLM-4.7-Flash’s architecture:

FlashAttention-4 produces NaN gradients for this model’s head_dim-256 partial-rotary attention; falling back to attn_implementation='sdpa'.

We found that the incompatibility only appears in the backward pass; the forward computation is correct. So we designed a hybrid scheme: FA4 handles the forward pass, while FA2 takes over for the backward. The hybrid approach gave a 1.36× increase in attention throughput.

We’ll have a separate post on FA4-fwd-FA2-bwd in our research blog; stay tuned! 💚

Kernels

GLM-4.7-Flash is a Mixture-of-Experts model. Tokens are distributed unevenly across experts, so the MLP blocks receive tensors of varying width. The Grouped-GEMM kernel batches matrix multiplications of varying shapes into a single kernel launch. Enabling it gave a 1.6× speedup on the FFN step.

Additionally, with Liger Fused Linear Cross-Entropy, we were able to compute loss without full logit matrix materialization in VRAM, preventing OOM on long sequences.

Together, these improvements yield up to a 1.63× training speedup over TRL, raising throughput to 11.1k tokens/s on a single B300 GPU on context length 4k:

Agentic infra challenge

Agentic evaluation at scale turned out to be a real infrastructure problem.

With thousands of concurrent agents, Docker brings high startup overhead and heavy disk pressure, and quickly saturates the local network — producing a high rate of environment startup errors.

Kubernetes struggled at high concurrency on a single host as well: it couldn’t set up and tear down pods fast enough, triggering timeouts. The problem was worst on powerful hosts: they could support enormous concurrency, but pod scheduling couldn’t keep up, so we never reached full CPU or RAM utilization.

Overlay sandboxes

Our solution is the Overlay Sandbox: instead of a container image, the agent runs on the bare host, inside an isolated filesystem and process group.

It has three layers: a base filesystem with the OS and repository, a hidden layer with the verifier suite, and an upper layer holding all edits and pip-installed packages.

Because overlay mounts are lazy (they don’t even index the filesystem!) and track only modified files, sandbox startup is essentially free.

Since SWE-rebench-V2 is built from public repositories, agents must run without network access — otherwise, a model can simply fetch a real fix from GitHub. Before restricting network, we caught Fable 5 and Sonnet 5 doing exactly this — retrieving the upstream patches directly on 20-40% of tasks.

Traffic recorder

Our sandbox also includes a traffic recorder — a tiny proxy that forwards requests to the provider (OpenRouter / vLLM / SGLang) and records every response. It’s a harness-agnostic way to assemble full trajectories in OpenAI format and collect performance metrics.

Evaluation

Every agent runs with max_turns = 200, medium reasoning effort, and default sampling parameters.

Inference costs are calculated from OpenRouter pricing tables. We report the theoretical price under the assumption of perfect caching — actual OpenRouter costs were higher due to inconsistent routing. For locally served GLM-4.7-Flash-Coder, we applied the per-request pricing of GLM-4.7-Flash via OpenRouter’s DeepInfra provider.

OpenRouter providers per model Prompt template

You are solving a real GitHub issue in the `{repo}` repository. The repo is already cloned and set up in the current working directory.

## Workflow

1. **Understand the issue.** Read the problem statement carefully. Identify the expected vs actual behavior.
2. **Explore the codebase.** Find the relevant files using grep, find, or by reading the project structure. Read the code that needs to change.
3. **Reproduce the bug** if possible — run the relevant test file or write a small script to confirm the issue exists.
4. **Implement a minimal fix.** Change only what is necessary. Do not refactor unrelated code, do not add features beyond what the issue asks for.
5. **Verify your fix.** Run the project's existing test suite to make sure you haven't introduced regressions. Look at the project's test configuration (Makefile, tox.ini, pyproject.toml, CI config) to find the right test command.
6. **Mark the task complete** only after you've verified existing tests pass with your changes.

## Issue

{problem_statement}

## Expected Interfaces

The following interfaces are expected by the test suite. Pay close attention to function names, parameter signatures, and return types — the tests verify these exactly.

{interface}

Inference with SGLang 0.5.14, CUDA 13

python3 -m sglang_router.launch_server \
  --model-path whitecircle/GLM-4.7-Flash-Coder \
  --trust-remote-code \
  --host 0.0.0.0 --port 30000 \
  --context-length 202752 \
  --prefill-attention-backend trtllm_mla \
  --decode-attention-backend triton \
  --page-size 64 \
  --tool-call-parser glm47 --reasoning-parser glm45 \
  --router-policy cache_aware \
  --tp-size 1 --dp-size 8

Team

• Project lead: Oleg Smirnov
• Agentic sandboxes: Konstantin Korolev, Sergei Bratchnikov
• Contributors: Mohamed Mekkouri, Dmitrii Kharlapenko, Daniil Oblakov

A White Circle project