Ling 3.0 Flash — CIRU IU4

Current runtime release: v1.1.1

Ciru builds for Kairic.ai to bring the best inference optimizations to production.

Ling-3.0-Flash-CIRU-IU4 is CIRU's Ling-specific INT4 checkpoint and AMD Strix Halo serving build for InclusionAI's 124B-parameter, 5.1B-active Ling 3.0 Flash model.

Custom runtime required for native IU4: use the ciru/ling3-gfx1151-iu4 branch of ciru-ai/vllm. Stock vLLM can load these weights through W4A16 compatibility, but it does not provide the IU4 path or reproduce the performance figures below.

This release combines three complementary changes:

• Better S4/G32 weights. CIRU recalibrated all 61,440 routed-expert projections in layers 2–41 using a Ling activation-importance objective. The checkpoint remains symmetric signed INT4, group size 32, in the same compressed-tensors format as the official model.
• Native S4 × S4 prefill on AMD IU4. Prefill uses a two-plane residual activation representation and the native gfx1151 integer WMMA lane. The packed checkpoint bytes remain canonical and safe for the normal W4 path.
• Production W4 decode with native MTP K1. Sparse decode stays on the optimized W4 path, where Ling's 512-expert/top-8 routing is better matched to small token batches.

The result is one production profile: native IU4 where it earns throughput, optimized W4 where sparse decode wins, and no user-facing mode selection.

Weight fidelity

Against the unchanged official INT4 checkpoint on CIRU's locked 92 × 4096-token BF16-reference corpus:

The recalibration changed weights, not the model architecture or tensor schema. The final next-token-prediction layer remains byte-identical to the official checkpoint because it was outside the calibration coverage.

Measured serving performance

Validated on one Ryzen AI MAX+ 395 / Radeon 8060S Strix Halo APU with 128 GB unified memory, native Linux, ROCm/HIP 7.15, Python 3.12, Torch 2.13 ROCm, Triton 3.8, and the CIRU gfx1151 vLLM runtime.

Six simultaneous 4K requests

Matched warmed run: six unique uncached 4096-token prompts, 256 forced output tokens each, MTP K1, 16K scheduler budget, prefix reuse disabled.

Both arms completed all requests with zero cached prompt tokens, zero prefix hits, zero preemptions, and no faults.

Dynamic long-context scheduling

The current runner automatically chooses the scheduler shape that fits the live Ling workload. One runnable request—or a request that began with more than 8K uncached prompt tokens—is held to the faster 8K prefill shape until its prompt completes. When two or more short/medium prefills are runnable, the budget rises to 16K. Decode keeps its existing priority.

Against the previous static-16K public launcher configuration, an order-balanced two-mirror warmed comparison on the same diagnostic source (policy disabled for control) at C1 with exactly 32K uncached input tokens and 128 output tokens produced:

At C6 with six 8K uncached prompts, the same mirrored comparison retained the static-16K path: pooled PP was −0.59%, median request TG was −0.25%, and wall time was +0.58%. All measured requests completed without cache reuse, preemptions, or device faults. Both C1 token sequences were exact; a natural HumanEval/0–5 check passed 24/24 executions across the four independent server launches.

The raw order-balanced matched rows were:

An earlier dedicated-profile oracle check is also retained. It compared the dynamic profile with separately tuned static 8K C1 and static 16K C6 servers, so it is not the deployed-default comparison above:

The dedicated static-8K oracle was 1.95% and 2.05% faster in C1 wall time at 16K and 32K. Dynamic C6 wall was +1.92% at 4K and +0.49% at 8K. These small oracle gaps do not justify a second serving profile.

Dynamic C1 context sweep

One excluded 512-token warm request was followed by one uncached measured pass at each context, with exactly 128 forced output tokens:

These are one-pass boundary measurements, not best-of results. The isolated 4K row did not reproduce after a same-concurrency warm batch: warmed C1/4K measured 700.276 PP tok/s in the ladder below. Both values are reported so the one-pass anomaly remains visible.

Dynamic 4K concurrency ladder

Each row used one excluded same-concurrency warm batch followed by one measured batch of unique uncached 4096-token prompts and exactly 128 output tokens per request:

All 55 measured requests completed with exact prompt/output accounting, zero cache reuse, zero preemptions, and no runtime faults. C8 is peak aggregate decode. C9 crosses from two to three 16K prefill waves. The launcher stays at six default slots because C8 improves end-to-end output by only 2.25% over C6 while increasing batch wall time by 30.4%.

Full-agent dynamic-build validation

The thinking-enabled HermesAgent-20 C6 run measured:

There were zero preemptions, zero device/runtime faults, and no scenario timeout. This is a standalone dynamic-build validation, not a matched comparison to an older run with different timeout or sampling conditions.

Full HumanEval and HumanEval+

The production profile completed a six-worker full-suite run with workers replenished continuously:

Sampling used InclusionAI's recommended live settings (temperature=0.6, top_p=0.95, top_k=20). The harness imposed a 4096-token ceiling per task; seven tasks reached that ceiling, so these are the measured capped scores rather than an extrapolated score.

Model details

Usage

The custom runner, gfx1151 kernel sources, extension builder, and production launcher are published in tools/ling_iu4. This patch release is pinned to tag ling-iu4-v1.1.1.

1. Build the runner

Ubuntu 24.04 / Debian-family setup:

sudo apt update
sudo apt install -y build-essential git libnuma-dev libdrm-dev libelf-dev \
  ninja-build pkg-config python3.12 python3.12-dev python3.12-venv xxd
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

git clone --branch ling-iu4-v1.1.1 \
  https://github.com/ciru-ai/vllm.git "$HOME/vllm-ling-ciru-iu4"
cd "$HOME/vllm-ling-ciru-iu4"
bash tools/ling_iu4/install_runtime.sh
source .venv/bin/activate

The installer rebuilds the wheel-pinned ROCr userspace library with the host compiler. This prevents the distributed ROCm 7.15 runtime from leaving a CPU core continuously busy while the gfx1151 server is idle. v1.1.1 activates the corrected library inside the private venv and installs an early-preload bootstrap; the launcher independently checksum-verifies and preloads it. The fix therefore remains active for the supported launcher, direct vllm serve, and python -m vllm starts.

Existing v1.0 or v1.1.0 installs can apply only the hardened idle-runtime activation without rebuilding vLLM, the IU4 extension, or the model:

git fetch origin ciru/ling3-gfx1151-iu4 --tags
git checkout ling-iu4-v1.1.1
VENV="${VIRTUAL_ENV:-$PWD/.venv}" \
  bash tools/ling_iu4/install_rocr_idle_fix.sh "$PWD"

Restart the server afterward. Once it is ready and has no active requests, verify both the library mapped by EngineCore and a fresh per-thread CPU sample:

bash tools/ling_iu4/verify_rocr_idle_fix.sh

The verifier reports ROCr mapping: PASS and Idle CPU: PASS when the fix is active. It samples current CPU use rather than the lifetime-average %CPU reported by some ps commands after compilation-heavy startup.

2. Download the model

hf download jcbtc/Ling-3.0-Flash-CIRU-IU4 \
  --local-dir "$HOME/models/Ling-3.0-Flash-CIRU-IU4"

3. Launch the production profile

From the CIRU vLLM checkout:

source .venv/bin/activate

MODEL_PATH="$HOME/models/Ling-3.0-Flash-CIRU-IU4" \
  bash tools/ling_iu4/run_ling_ciru_iu4.sh

The launcher provides the measured configuration: native 262,144-token context, six scheduler slots, a dynamic 8K/16K Ling batched-token policy, chunked prefill, prefix caching, Ling tool/reasoning parsers, optimized W4 sparse execution, native IU4 execution for occupied expert work, and MTP K1. MAX_NUM_BATCHED_TOKENS remains the upper bound. Override HOST, PORT, MAX_MODEL_LEN, MAX_NUM_SEQS, MAX_NUM_BATCHED_TOKENS, or GPU_MEMORY_UTILIZATION only when needed.

To use a conventional static budget for diagnosis or a deliberately fixed deployment, disable the policy explicitly:

VLLM_LING_DYNAMIC_BATCHED_TOKENS=0 MAX_NUM_BATCHED_TOKENS=8192 \
  MODEL_PATH="$HOME/models/Ling-3.0-Flash-CIRU-IU4" \
  bash tools/ling_iu4/run_ling_ciru_iu4.sh

The kernel build and arithmetic smoke were reproduced from the clean public branch on CIRU's Radeon 8060S (gfx1151). The performance measurements were validated on CIRU's native-Linux NixOS host. The Ubuntu commands above are the supported mainstream build path, but are not presented as a separate clean-host benchmark validation. Native Windows vLLM is unsupported; Windows users require WSL2 with AMD ROCDXG.

Request example

curl -fsS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Ling-3.0-Flash-CIRU-IU4",
    "messages": [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
    "chat_template_kwargs": {"enable_thinking": true},
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 1024
  }'

Intended use and limitations

This release targets local, OpenAI-compatible text generation and coding-agent workloads on high-memory AMD Strix Halo systems. It preserves Ling's native reasoning and tool-call behavior.

• The native IU4 acceleration is hardware-specific to gfx1151; other devices use their supported W4A16 kernels.
• The checkpoint is approximately 77 GB before runtime caches and KV cache. A 128 GB unified-memory system is the validated target.
• Benchmark results are single-host measurements and should not be generalized to different ROCm, vLLM, firmware, power, or memory configurations.
• Generated code must be reviewed and sandboxed before production execution.

Credits and license

• Base model, tokenizer, architecture, and official INT4 checkpoint: InclusionAI, MIT license.
• Inference engine: vLLM, Apache-2.0.
• AMD ROCm, HIP, and gfx1151 integer WMMA support: AMD and their respective upstream projects.
• Weight calibration, native IU4 kernels, runtime integration, profiling, and validation: CIRU.

The original MIT license governs the model weights and accompanying model code. Review the upstream model card before deployment.