Ling-3.0-tiny ONNX — Q4 WebGPU + CUDA

Community ONNX conversion of inclusionAI/Ling-3.0-tiny, a 7.9B-total / 1.3B-active-parameter hybrid reasoning MoE model.

[!IMPORTANT] This is an independent community conversion, not an official inclusionAI release. Read the original model card for training, evaluation, intended use, and base-model limitations.

The WebGPU layout follows the standard Transformers.js q4f16 contract used by browser-oriented ONNX repositories: the graph is under onnx/, external tensor data is split into three sub-2 GB files, activations and cache are FP16, and hybrid recurrent-cache names use the Qwen3.5-compatible convention already supported by Transformers.js.

Run in the browser with WebGPU

Install Transformers.js 4.2 or newer:

npm install "@huggingface/transformers@^4.2.0"

import { pipeline, TextStreamer } from '@huggingface/transformers';

const generator = await pipeline(
  'text-generation',
  'webbrain-one/Ling-3.0-tiny-ONNX',
  {
    device: 'webgpu',
    dtype: 'q4f16',
  },
);

const messages = [
  { role: 'user', content: 'Explain why the sky is blue in two sentences.' },
];

const output = await generator(messages, {
  max_new_tokens: 128,
  do_sample: false,
  tokenizer_encode_kwargs: { enable_thinking: false },
  streamer: new TextStreamer(generator.tokenizer, {
    skip_prompt: true,
    skip_special_tokens: true,
  }),
});

console.log(output[0].generated_text.at(-1)?.content);

For thinking mode, set enable_thinking: true. The original model card recommends temperature: 1.0, top_p: 0.95, and top_k: 20 when sampling in thinking mode.

Browser requirements

• A current desktop browser with WebGPU enabled; Chrome or Edge is recommended.
• Approximately 4.85 GB of model downloads on first load, plus browser cache.
• Enough GPU memory for all weights, runtime buffers, state, and the requested context. The 1.3B active-parameter figure reduces compute, but all 7.9B model parameters still need to be stored.
• Start with a short prompt and modest max_new_tokens, then increase context after confirming memory use on the target device.

The validation machine had two discrete NVIDIA GPUs, and Chromium selected the display-connected adapter. On multi-GPU systems, check the adapter selected by the browser rather than assuming it will match a CUDA compute workload.

This graph depends on WebGPU implementations of MatMulNBits, QMoE, LinearAttention, CausalConvWithState, and GroupQueryAttention. It is not a WASM/CPU fallback model.

Use from WebBrain

Use the same repository ID and standard Transformers.js settings:

model: webbrain-one/Ling-3.0-tiny-ONNX
device: webgpu
dtype: q4f16
task: text-generation

WebBrain should select onnx/model_q4f16.onnx and fetch the three external-data files declared by config.json.

Run the CUDA variant with Python

The repository also retains the separately validated CUDA-targeted ONNX Runtime GenAI graph.

pip install "onnxruntime-gpu>=1.28.0" \
  "onnxruntime-genai-cuda>=0.15.2" \
  "transformers>=4.57,<5"

import numpy as np
import onnxruntime_genai as og
from transformers import AutoTokenizer

model_dir = "Ling-3.0-tiny-ONNX"
model = og.Model(model_dir)
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)

input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain why the sky is blue."}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="np",
    enable_thinking=False,
)

params = og.GeneratorParams(model)
params.set_search_options(
    max_length=int(input_ids.shape[-1]) + 128,
    do_sample=False,
)

generator = og.Generator(model, params)
generator.append_tokens(np.asarray(input_ids[0], dtype=np.int32))
prompt_length = int(input_ids.shape[-1])

while not generator.is_done():
    generator.generate_next_token()

print(tokenizer.decode(generator.get_sequence(0)[prompt_length:], skip_special_tokens=True))

Architecture and quantization

• 18 Kimi Delta Attention layers with recurrent and convolution state
• 6 Multi-Latent Attention layers with KV cache
• 23 sparse MoE layers with 128 routed experts, top-8 group-limited routing, expert bias, and one shared expert
• one dense MLP layer at the start of the decoder
• symmetric Q4/block-32 dense and routed-expert weights
• FP16 embeddings, activations, recurrent state, and KV cache
• FP32 MoE router weights and routing math

The ONNX graph contains 235 MatMulNBits, 23 QMoE, 18 LinearAttention, 18 CausalConvWithState, and 6 GroupQueryAttention nodes.

Validation

• The complete CUDA artifact generated successfully on an NVIDIA GeForce RTX 5090 with ONNX Runtime GenAI 0.15.2 and ONNX Runtime GPU 1.28.0.
• The WebGPU repack was verified tensor-for-tensor against the CUDA graph; all 4,835,749,912 external tensor bytes are identical.
• Transformers.js 4.2.0 loads the config, dispatches Qwen3_5ForCausalLM, finds all 18 recurrent/conv caches and 6 attention caches, and applies the original Ling tokenizer/chat template.
• A full remote-repository test passed in Chrome 150 with Transformers.js 4.2.0 on an NVIDIA T400 4GB WebGPU adapter using Windows shared-memory oversubscription. First load/session creation took 925.87 seconds; a deterministic 16-token generation took 90.18 seconds and produced a coherent answer. This is a compatibility smoke test, not a performance benchmark.

Limitations

• Q4 quantization can change outputs and quality relative to the original BF16 checkpoint. No benchmark parity claim is made here.
• Browser support, GPU limits, shader compilation time, and memory behavior vary by operating system, browser version, and GPU driver.
• The advertised 131,072-token context is architectural; practical browser context is limited by available GPU memory.
• The WebGPU config uses Transformers.js's existing qwen3_5_text hybrid-cache adapter solely as a runtime compatibility layer. The underlying graph and weights remain Ling/Bailing Hybrid, preserved in config_bailing_original.json.

Attribution and license

The model architecture, checkpoint, tokenizer, and chat template are by inclusionAI. This conversion retains the base model's MIT license. Please cite and credit the original Ling-3.0-tiny release when using or redistributing this artifact.