Holo4-35B-A3B EXL3 6 bpw with donor MTP

Holo4-35B-A3B language and vision weights from groxaxo/Holo4-35B-A3B-exl3-6bpw, with a quantized MTP draft component extracted from groxaxo/Qwen3.6-35B-A3B-Heretic-exl3-4bpw.

This is an experimental component graft, not an official H Company MTP release. The current Holo4 source has no MTP weight tensors. Its main language weights, output head, vision encoder and tokenizer vocabulary are retained. No training was performed. The donor MTP component comes from llmfan46's native-MTP-preserved Qwen3.6 Heretic model, and is used to propose tokens that the Holo4 target verifies.

The Holo4 language model uses a converter-reported 6.08 bpw, its output head uses 8 bits, the vision encoder uses 6 bits, and the donor MTP component uses 4 bits. The donor contains 3,113 stored quantized tensors (427,131,936 bytes) representing one native MTP layer. The Holo4 and donor token vocabularies and architecture dimensions were checked for compatibility.

Use an EXL3 runtime with Qwen3.5 MoE MTP support. The example inference script loads the target and donor component from this checkpoint. The tested deployment uses two RTX 3090 GPUs, layer splitting, Q8 target and draft KV caches, and three draft tokens. Runtime revision and validation evidence are in provenance.json and validation.json. The tested runtime includes local changes; operation on an unmodified upstream runtime or other hardware is not claimed.

The stock Holo4 chat template and native 262,144-token context configuration are included. The deployed long-context profile uses YaRN factor 2, a 520,000-token request limit, two concurrent requests, and a shared 1,040,384-token cache. See serving-profile.json and validation.json for the checks actually performed.

Initial short text tests measured 45-80% draft acceptance and 1.2-1.7x observed speedup across three prompts. Greedy outputs differed from the no-MTP baseline; these small measurements do not establish lossless numerical equivalence or an accuracy benchmark. No Holo4 computer-use or GUI benchmark scores are claimed for this graft.

Original Holo4 model by H Company. Donor source by llmfan46, based on Qwen/Qwen3.6-35B-A3B. Apache-2.0 licence included. See provenance.json for exact source files and attribution.

Large-context prefill on the tested two RTX 3090 GPUs can take tens of minutes. The deployed request timeout is 7,200 seconds.

Run the text example with python example_inference.py /path/to/checkpoint. To allocate the deployed long-context cache, add --context 520000 --batch-size 2 --yarn-factor 2. Set CUDA_VISIBLE_DEVICES to two suitable GPUs.