MOSS-Image Qwen3.5-9B text encoder, INT8 (W8A8, ConvRot)
The Qwen3.5-9B text encoder of MOSS-Image
in one 9.2 GB file instead of the 16.8 GB bf16 shards, in ComfyUI's mixed-precision
int8_tensorwise + convrot format. Its matmuls run in int8 (weights and activations),
about 1.7× the encode speed of the fp8 file
at the same VRAM and nearly the same error.
| Files | qwen35_9b_int8_convrot.safetensors (9.20 GiB), config.json (the Qwen3.5-9B config, for transformers) |
| Quantized | the 248 decoder-layer linears (attention, gated-DeltaNet projections, MLP): each weight rotated per 256-column group by a Hadamard matrix (ConvRot), int8 with one fp32 weight_scale per output row |
| Kept as-is | token embedding, GDN conv1d / A_log / dt_bias, the vision tower |
| Changed | the decoder layers' RMSNorm weights (fp32) carry SmoothQuant scales (below); they belong to this file's linears and don't mix with the bf16 ones |
| Matmul | W8A8: the input is rotated by the same Hadamard and rounded to int8 per token (comfy_kitchen.int8_linear). The weight can also be dequantized for a bf16 matmul (weight-only) |
anima_lora
The default text encoder for MOSS caching and generation (text_encoder=int8,
--text_encoder int8): fetched on first use (make download-model ARGS=moss-te-int8),
all 32 decoder layers stay on a 16 GB card. The tokenizer still comes from MOSS-Image-Base.
text_encoder=bf16 / fp8 select the reference shards or the fp8 file.
ComfyUI
Put the file in models/text_encoders/ and load it with MOSS-Image Load Qwen3.5
(the moss-image node); ComfyUI's mixed-precision loader reads comfy_quant per layer.
Measured
RTX 5070 Ti 16 GB, against the bf16 encoder. Speed and VRAM (bench/moss_te_fp8, 34 prompts,
same format and kernel):
| bf16 | fp8 (weight-only) | INT8 W8A8 | |
|---|---|---|---|
| Encode, tokens/s | 2250 (2 layers paged) | 1467 | 2459 |
| Peak VRAM, decoder layers resident | 12.59 GiB | 6.69 GiB | 6.54 GiB |
Embedding relative L2 error on the held-out split of the calibration set (121 entries; image rows are manga / colorize pages through the vision tower):
| fp8 GPTQ | INT8 (this file) | |
|---|---|---|
| tags | 0.039 | 0.042 |
| position clauses | 0.040 | 0.046 |
| natural language | 0.044 | 0.043 |
| layout JSON | 0.039 | 0.047 |
| OCR captions | 0.051 | 0.053 |
| JA / KO / ZH | 0.054 | 0.053 |
| page + caption (image rows) | 0.10 | 0.10 |
| all | 0.095 | 0.097 |
Plain INT8 GPTQ (no smoothing) reads 0.105 overall in the same W8A8 mode; round-to-nearest is worse again.
How it was made
project/moss_lora/src/te_gptq.py --grid int8_convrot --smooth 0.5 --act_aware in
anima_lora, on MOSS text-encoder inputs (tag
captions, layout JSON, English and JA/KO/ZH prompts, manga and colorize pages through the
vision tower; te_calib.py), layer by layer:
- SmoothQuant (α 0.5): per input channel, activation outliers are moved into the
weight; the scales are folded into the preceding RMSNorm (
(1 + w) / s − 1) and, fordown_proj, intoup_proj's rows. - Activation-aware GPTQ: the Hessian comes from the rotated, int8-rounded inputs the W8A8 kernel actually sees, and the rounding targets the least-squares weight for those inputs. Each layer is calibrated on the previous layers' W8A8 outputs.
License
Apache-2.0, as MOSS-Image and Qwen3.5. Weights derived from
OpenMOSS-Team/MOSS-Image-Base text_encoder/.