Nemotron 3.5 Lightning 30B-A3B Ternary, OpenVINO, Intel Arc A770

Update 2026-11-01: 95.9 tok/s (model 2y7 + plugin build G)

• model/nemotron_2y7.xml uses the same weights as 2y. Load it with core.read_model("model/nemotron_2y7.xml", "model/nemotron_2y.bin").
• 2y7 updates the Mamba states in place. It needs the included plugin DLL (build G = build E + build_F.diff + build_G.diff). On a stock OpenVINO GPU plugin 2y7 gives wrong output; use 2y there.
• Measured on Arc A770 16 GB, decode median: 2y7 10.43 ms (95.9 tok/s), 2y 10.94 ms on the same plugin. Greedy 30/30 on p1, p7, p64 vs reference; p300 same as 2y.
• Use a new, empty CACHE_DIR whenever you change the plugin DLL.
• The previous release (2y + build E, 91.30 tok/s) is kept under tag v2y-E.

NVIDIA's Nemotron 3.5 Lightning 30B-A3B (hybrid Mamba-2 + MoE + attention, 128 experts, 6 active) in nota-ai's ternary weights, running on one Intel Arc A770 16 GB through OpenVINO with custom OpenCL kernels. No speculative decoding, no prompt lookup: one forward pass per generated token.

Measured speed (Arc A770 16 GB, Windows, batch 1, greedy)

Needs the patched GPU plugin in plugin/ (build E: build_D.diff + build_E.diff on OpenVINO 2026.4.1).

Logs: logs/log_eval_2yE.txt. Size and sha256 of the model, ops, scripts, plugin and logs: MANIFEST.txt.

Correctness

Checked against an f16 reference graph of the same weights (scripts/eval_nemo.py):

• 3 test prompts (1, 7 and 64 tokens): same top-1 token, first 30 greedy tokens identical (30/30).
• 300-token prompt: same top-1 token, logit cosine 0.99985; the greedy sequences split after 8 tokens at a near-tie. Average log-prob over the prompt: -0.659 (reference -0.655).
• Prompt processing in chunks gives the same top-1 token as one pass at every tested length up to 16k.

These are numeric checks against the reference, not a quality benchmark of the model itself.

What is in this repo

Requirements

• Intel Arc A770 16 GB (other Arc cards untested), Windows.
• Python with openvino==2026.4.1 (see requirements_nemo_D.txt).
• The patched GPU plugin: replace openvino_intel_gpu_plugin.dll in that OpenVINO install's site-packages\openvino\libs\ with plugin/openvino_intel_gpu_plugin.dll. Use a separate venv for this. The patches fix custom-layer model caching and per-layer kernel recompilation in the stock plugin (reported upstream as openvinotoolkit/openvino#39296 and #39297).
• The tokenizer is not included: use the one from the base model.

Running it

scripts/eval_nemo.py shows the full loop. In short:

import sys, numpy as np, openvino as ov
sys.path.insert(0, "ops")
import os; os.environ["MOE_UP_VARIANT"] = "g"   # the variant used for the measured numbers
import ops_2p

core = ops_2p.register(ov.Core(), out_dir="kernels")
cm = core.compile_model("model/nemotron_2y.xml", "GPU", {"CACHE_DIR": "cache", "PERFORMANCE_HINT": "LATENCY"})
req = cm.create_infer_request()
req.reset_state()

def feed(ids, pos):                       # ids: list of up to 256 token ids starting at position pos
    M = len(ids)
    W = pos + M                           # mask width: exactly past + new tokens
    return {"input_ids": np.array([ids], np.int64),
            "causal_mask_exact": np.where(np.arange(W)[None, :] > (pos + np.arange(M))[:, None], -10000.0, 0.0).astype(np.float16)[None, None],
            "kv_write_indices": pos + np.arange(M, dtype=np.int64),
            "mamba_mask": (np.arange(M)[None, :] > np.arange(M)[:, None])[None, None]}

# call req.reset_state() before each new prompt
# prompt in chunks of <= 256 tokens, then one token per step:
# req.infer(feed(chunk, pos)); logits = req.get_output_tensor(0).data[0, -1]

The first compile takes about 5 minutes; later runs load from CACHE_DIR in seconds. No chat runner or server is included yet.

How it got this fast

Main steps, each measured: custom OpenCL ops for the Mamba-2 and MoE layers that read only the 6 routed experts' 2-bit weights; attention over the filled KV slots only, with the window size taken from the mask's shape so the CPU never waits for the GPU; the output head kept at the GGUF's Q8_0 precision instead of f16; and MoE down-projection kernels split across more GPU threads. Starting point on the same card: 36.5 tok/s generation, 286 tok/s prompt.

Credits and license

• Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, OpenMDW-1.1.
• Ternary weights: nota-ai/Nemotron-3.5-Lightning-30B-A3B-Nota-Ternary-MLX-1014, OpenMDW-1.1.
• This conversion and the model weights here: OpenMDW-1.1 (LICENSE). OpenVINO and the patched plugin: Apache-2.0 (plugin/LICENSE.OpenVINO).

Made for MustacheOS, a new independent project building a fully local AI platform and a future community where people and AI models meet and talk.

Work in progress: more kernel work is underway, and updates will be posted here with their measurements.