AliceAI-T5-35B-A0.6B-NVFP4

Selective post-training NVFP4 quantization of Yandex AliceAI-T5-35B-A0.6B, built by ProCreations.

20.09 GB of checkpoint weights; 3.44× smaller than the source safetensors. No QAT, fine-tuning, or distillation was performed.

This is an encoder-decoder base model, with the upstream UL2-style input format. It is not an instruction-tuned chat model.

Quantization recipe

• 28/28 grouped MoE layers use packed NVFP4 expert weights.
• E2M1 values are packed two per byte, with one E4M3 scale per 16 weights and a FP32 global scale per expert matrix.
• Weight scales minimize activation-weighted squared error across 21 neighboring representable E4M3 candidates. The candidate set includes conventional max scaling. Per-expert input-energy estimates use 10% shrinkage toward uniform weighting for stability.
• This scale search reduced the recorded activation-weighted weight reconstruction error by 34.72% relative to max scaling with the same global scales; this is a fitting metric, not a downstream accuracy score.
• Activation block scales are computed dynamically during inference. The first expert matrix also uses a dynamic per-forward global scale. The second matrix uses a per-expert global scale calibrated with 4× headroom.
• Attention projections, routers, shared embeddings, LM head, normalization, and biases remain at their loaded higher precision.
• A bounded development-set search can restore up to four sensitive MoE layers to BF16. Actual restored layers: No additional expert layers were retained in BF16.
• Calibration used 112 samples: 48 English Wikipedia continuations, 48 Russian Wikipedia continuations, and 16 Alpaca instruction/answer examples. Dataset revisions and selected sample identities/hashes are in data_manifest.json.
• Source revision: a0d71f58c40d6affe461797b30b35ff47f52a5f2. Seed: 20260914.

“Dynamic” here describes dynamic activation block scaling and selective precision allocation. This is an independent recipe, not Unsloth Dynamic V3 or an official NVIDIA/Yandex release.

Measured quality

Paired BF16/NVFP4 next-token evaluation uses separate Wikipedia articles, a 192-token encoder prefix, and 64 teacher-forced continuation tokens. The 32-sample development set selected precision; the 64-sample test set was held out from calibration and selection.

Release thresholds were set before evaluation: combined NLL increase ≤0.12, per-language increase ≤0.16, and per-language top-1 agreement ≥80%. The saved checkpoint was reloaded and compared against the in-memory candidate on eight test samples.

These are small continuation-fidelity checks. They do not establish preservation of the upstream MMLU, mathematical reasoning, coding, or 128K-context benchmark scores. No such scores are claimed.

Additional paired checks

A separate 228-question MMLU sample uses five examples from each subject's development split and four randomly selected test questions from each of 57 subjects. The identical prompts are scored using A/B/C/D next-token logits in the model's UL2 encoder-decoder format. This sample was not used to tune the quantization.

Paired changes: 9 correct→incorrect, 5 incorrect→correct. A maximum five-percentage-point drop was specified before this check. Results, question indices, prompt hashes, and choice scores are in mmlu_sample.json. This small subject-balanced subset is not a full official MMLU result and cannot resolve small accuracy differences reliably; its prompting may differ from Yandex's internal harness.

The eight-prompt generation check in generation_smoke.json includes wrong and incomplete answers from both the BF16 source and the quantized model. For example, both answered the zero-shot Russian multiplication prompt incorrectly; the water-formula prompt produced H in BF16 and a blank response in NVFP4. This base model should not be treated as a ready-to-use chat assistant.

Measured speed

Hardware: NVIDIA RTX PRO 6000 Blackwell Workstation Edition. Same Transformers attention implementation, batch size 1, prompt bytes, greedy generation, and 64-token output length. One warm-up and three timed repetitions. Timing includes the encoder and decoder prefill.

Observed end-to-end speed ratio: 3.59×. This single-prompt result is not a general throughput guarantee. Output text and raw timings are in evaluation.json.

This speed comparison includes the change from the upstream BF16 grouped-matrix runtime to fused FlashInfer MoE kernels, as well as the lower precision. Both runs force 64 generated tokens for matched work, even if the model would otherwise stop earlier.

Runtime and compatibility

The checkpoint includes custom Transformers model code and runs native FlashInfer CUTLASS NVFP4 kernels on NVIDIA Blackwell. The expert weights remain packed on the GPU; they are not expanded to BF16 on load.

Use AutoModelForSeq2SeqLM, trust_remote_code=True, and torch.bfloat16. CUDA, a compatible NVIDIA driver, and the FlashInfer build requirements are necessary. Kernel compilation may make the first call slow.

Validated packages: PyTorch 2.10.0+cu130, Transformers 5.14.1, FlashInfer 0.6.18.post1, Accelerate as specified in the recipe requirements. Tested hardware is listed above. Other GPUs and package versions are untested.

This custom architecture/packed format is not a drop-in vLLM, SGLang, llama.cpp, or standard ModelOpt checkpoint. Those runtimes need an AliceAI-specific integration. CPU/MPS inference and training of the packed checkpoint are not supported.

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "ProCreations/AliceAI-T5-35B-A0.6B-NVFP4"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16,
    device_map={"": "cuda:0"}, attn_implementation="eager",
).eval()

prompt = "Question: What is the capital of France?\nAnswer:"
mode, span = tokenizer.convert_tokens_to_ids(["[_S_]", "<SPAN#0>"])
content = tokenizer(prompt, add_special_tokens=False).input_ids
input_ids = torch.tensor([[mode, *content, span]], device="cuda")
decoder_input_ids = torch.tensor([[model.config.decoder.bos_token_id, span]], device="cuda")
with torch.inference_mode():
    output = model.generate(
        input_ids=input_ids, attention_mask=torch.ones_like(input_ids),
        decoder_input_ids=decoder_input_ids, do_sample=False, max_new_tokens=64,
    )
print(tokenizer.decode(output[0, 2:], skip_special_tokens=True).split("<SPAN#", 1)[0])

Reproduce and inspect

See reproduce/quantize_alice.py, reproduce/nvfp4_kernels.py, data_manifest.json, build_environment.json, and evaluation.json. Quantization needs enough GPU memory for the BF16 source plus temporary tensors and enough CPU RAM to retain the BF16 fallback layers. The production loader is nvfp4_runtime.py.

Attribution and license

Original model and model code: Copyright 2026 YANDEX LLC, Apache-2.0. Upstream LICENSE and NOTICES are retained. Changes to the custom model files are marked in their headers. ProCreations' added quantization/runtime code is provided under Apache-2.0. FlashInfer is a separate Apache-2.0 dependency.

Recipe inspiration: NVIDIA Qwen3.8-Flash-Next-NVFP4, NVIDIA GLM-5.3-Flash-NVFP4, and Unsloth Qwen3.8-27B-NVFP4. Their benchmark results and runtime support do not apply to this model.