Average score (%) over 18 benchmarks by domain. The repository's README lists the benchmarks.
| Domain | Score |
|---|---|
| Knowledge and reasoning | 71.6 |
| Math | 50.5 |
| Code | 53.0 |
| Instruction following | 55.0 |
| Average | 62.8 |
vllm serve McGill-NLP/Nemotron-3.5-Lightning-30B-A3B-SFT-EchoMoE-teacher \
--data-parallel-size 8 --enable-expert-parallel \
--max-model-len 32768
The bf16 weights fit on one 80 GB GPU.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "McGill-NLP/Nemotron-3.5-Lightning-30B-A3B-SFT-EchoMoE-teacher"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained(repo)
messages = [{"role": "user", "content": "Give me a short introduction to large language models."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
With transformers, this model's log-probabilities differ from vLLM's more than for the other EchoMoE models. Without the mamba-ssm kernels installed, transformers 5.8.1 runs the Mamba-2 layers in plain PyTorch. On ARC-Challenge (5-shot) this model scores 0.9 points below its vLLM score, and the EchoMoE student scores 2.3 points below. Use vLLM to reproduce the reported numbers.
@misc{liu2026communicate,
title = {Communicate the Change: Efficient Expert Parallelism via Activation Compression},
author = {Liu, Fengyuan and Gala, Jay and Reddy, Siva and Bahdanau, Dzmitry},
year = {2026},
eprint = {2611.22215},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.22215},
}
This model is released under the OpenMDW License 1.1, the license of its base model, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16. The LICENSE file keeps NVIDIA's copyright notice.
model-00014-of-00016.safetensors 4.0 GBmodel-00003-of-00016.safetensors 4.0 GBmodel-00012-of-00016.safetensors 4.0 GBmodel-00009-of-00016.safetensors 4.0 GBmodel-00005-of-00016.safetensors 4.0 GBmodel-00007-of-00016.safetensors 4.0 GBmodel-00015-of-00016.safetensors 4.0 GBmodel-00011-of-00016.safetensors 4.0 GBmodel-00010-of-00016.safetensors 4.0 GBmodel-00008-of-00016.safetensors 4.0 GBmodel-00002-of-00016.safetensors 4.0 GBmodel-00006-of-00016.safetensors 4.0 GBmodel-00013-of-00016.safetensors 4.0 GBmodel-00004-of-00016.safetensors 4.0 GBmodel-00001-of-00016.safetensors 4.0 GBmodel-00016-of-00016.safetensors 3.2 GB