ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B or 8B

Projection matrices that let a small Qwen3-VL replace the Qwen3-VL-32B text encoder of MiniMax H3.

15.7 GB → 4.5 GB of VRAM, with no change to the diffusion model, the VAEs or the sampler.

---

v3.1 — better multilingual speech

No measurable gain in image over v3 — the whole gain is in speech. The image scores of the two generations overlap entirely, and nothing measured here separates either of them from the 32B, which is not the same as saying they are identical to it: the cosine is blind to countable attributes. What v3.1 improves is pronunciation in the ten languages that are not English — 29 %% fewer phoneme errors overall, 60 to 74 %% fewer in Spanish, French, German and Italian.

Eleven languages, 88 seconds. For each one the smallest file that matches the 32B — not the best one. Nine of the eleven run on a 4B. Direct link

⚠️ The video looks and sounds rough, on purpose. 0.3 MP, 6 sampling steps — the settings that made 297 renders affordable — then upscaled. The audio is tinny for the same reason, identically so on the 32B, with the same 19 dB dip between 1 and 3 kHz. This is a pronunciation test, not a showcase: it exists to let you hear which words come out.

Four new files: mmh3-4b-ClipProj-v3.1, mmh3-4b-ClipProj-v3.1-mlp, mmh3-8b-ClipProj-v3.1, mmh3-8b-ClipProj-v3.1-mlp. They load on node 0.1.13 with no code change — same base as v3. Start with mmh3-4b-ClipProj-v3.1: 26 MB of projection, 4.6 GB with its encoder.

What changed: the calibration corpus now gives every writing system a comparable share instead of being overwhelmingly English, with raw Arabic text added. Phoneme errors drop 29 % overall, 60 to 74 % on the European languages.

What it is measured against: the 32B does not reproduce itself. Change nothing but the seed and it re-pronounces 5.8 phonemes out of 75 differently. The four v3.1 files sit at 6.4 to 7.0 — so swapping the encoder costs about what re-rolling the seed costs. Every figure is normalised against that variance rather than against zero, and on that scale nothing separates 4B from 8B, or ridge from residual.

Full report — three seeds, 297 speech renders, 405 image renders, method, limitations and all the raw data: bench3.1/README.md

---

⚠️ Proof of concept — working, but a proof of concept. It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.

Arriving from a tutorial or an article? Anything published before 11 August names files that have moved. h3_qwen3vl_4b_tap24, h3_control_zero and h3_control_identity are still here, one folder down in obsolete/, so nothing is lost. But take the current set instead: mmh3-4b-ClipProj-celeb-mlp for a Qwen3-VL-4B, mmh3-8b-ClipProj-celeb-mlp for an 8B. They are better on every measurement below, and they need node 0.1.4 or later.

These files are useless on their own. They require the custom node: github.com/nicolab28/ComfyUI-ClipProj

Where this came from

I am not an ML researcher. I work in imaging, and programming is a tool and a hobby rather than my trade. This started as something to tinker with: I wanted to understand how a diffusion model actually uses its text encoder, and the only way I know how to understand something is to take it apart and see whether it still runs afterwards.

So the question was never "how do I save VRAM". It was "is this even possible at all". I expected it to fail. A linear map between two models that were never trained together, fitted in a single pass with no gradients and no learning rate, has no business producing usable video.

It did, and the first results were good enough that keeping them on my own disk seemed silly. That is the whole story, and it is why this is labelled a proof of concept rather than a tool: it was never designed as one.

It is also why there are so many measurements on the model card. Before showing this to anyone I had to convince myself I was not fooling myself, and most of what I tried along the way turned out to be wrong. Those attempts are written down as well, in MEASUREMENTS.md and CALIBRATION.md.

Update to node 0.1.4, and re-download the -mlp matrices

The -mlp matrices are now fp16 and half the size. The residual network was published in fp32 and the node forced fp32 on load regardless of the file, so storing it in half precision would have halved the download and saved nothing at all in VRAM. Node 0.1.4 keeps a residual in whatever precision it was saved in, converting its inputs and outputs around it instead. Measured: 240 MB on the card instead of 480 for the 4B, 288 instead of 576 for the 8B. The files here have been replaced under the same names — re-download them, and take 0.1.4 with them, because an older node will load them and cast them straight back up to fp32.

Node 0.1.4 also frees the card before loading a replacement encoder rather than after, which matters if yours is tight enough that two encoders will not sit on it at once.

Also in 0.1.3

Two reasons, one of them silent.

The -mlp matrices carry a residual network, and an older node ignores it without saying so. It reads the matrix, finds keys it does not know, drops them, and applies the linear part alone. Nothing fails, nothing warns, and you end up judging the plain matrix while believing you tested the residual. Node 0.1.3 reads them.

Everything is renamed. The old h3_qwen3vl_* files have moved to obsolete/ and the .pt copies are gone: opening a pickle executes code, which makes no sense for a file holding six tensors. If a workflow of yours names an old file, either point it at obsolete/ or, better, switch to the new set.

What this is

MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a [seq, 5120] tensor. This repository provides a learned map that lets a much smaller Qwen3-VL produce the same conditioning:

cond = ((h - mean_in) / std_in) @ W * std_out + mean_out

and, in the -mlp files, plus the output of a small residual network fed the same standardised input.

It works because every Qwen3-VL shares the same tokenizer (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. The matrix is fitted by plain ridge regression — no gradients, no epochs, no learning rate. The residual network is the only part that is trained.

Files

Put them in ComfyUI/models/clip_projections/.

Start with mmh3-8b-ClipProj-celeb-mlp if you have the VRAM, mmh3-4b-ClipProj-celeb-mlp otherwise.

All eight are calibrated on the same general corpus and measured on the same held-out prompts, so the column is comparable across every row.

Every matrix works on any variant of its own size: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on. The 8B matrices need an 8B encoder though — 4096 input dimensions instead of 2560 — and the node checks the width and refuses a mismatch.

Named people

This is what changed in 0.1.3, and it was a corpus problem.

The calibration corpus named a person on about 70 lines out of 8632, roughly 0.02 % of the training tokens. The directions of the hidden space that carry an identity were therefore constrained by nothing at all, and the fit put whatever minimised the error on landscape descriptions there. Named people came out as somebody else.

The -celeb matrices add 500 people, ranked by popularity, with five short prompts and two long ones each. What it buys and what it costs:

Seven thousandths of cosine on the general corpus, for six points on the tokens that carry an identity. The rest of the sentence improves too, because the celebrity prompts are short and the general corpus had nothing under fifteen words.

Two findings that decide how far this is worth pushing.

Two contexts per person are enough. Measured on contexts held out for people the matrix had seen: 0.9875 at two, 0.9945 at five, 0.9986 at twenty. Forty is a waste.

Five hundred names generalise to names never seen. A held-out band at popularity ranks 501 to 540, absent from every calibration, reconstructs at 0.8795 against 0.8844 for the covered ones. Covering 500 people does not teach 500 names; it teaches the map how to handle that region of the space. Going to several thousand would buy very little.

What still fails is not the corpus. Characters whose identity is a mask rather than a face come out as a stranger wearing the right costume. People whose fame predates the era when everything was photographed come out wrong or generic. And some names fail on the plain 32B too, so run the reference before blaming the projection — that check has overturned three of my own conclusions.

Where the calibration data comes from

The general corpus is GokuScraper/seedance-2-prompts-datasets, filtered to prompts of fifteen words or more and deduplicated: 8632 lines, median 128 words. The 500 named people come from a TMDB export published on Kaggle, ranked by popularity, with transliterated names dropped beyond rank 1000.

Around each name, five short prompts are generated from templates, and two longer ones in MiniMax H3's section format are written by Mistral Small and Gemini Flash Lite, half each. Everything needed to rebuild the corpus is in the node's calibration/ folder, including the system prompt the long prompts were written from.

The residual network

The -mlp files carry a d_in → 16384 → 5120 network with a GELU, added to the matrix rather than replacing it. Its last layer is initialised to zero, so at the first step the model reproduces the matrix exactly and can only improve on it. It is worth 0.05 to 0.08 of cosine, four times what multiplying the corpus by eleven buys the linear map.

Which of the two renders better is not settled. The cosine does not predict it — that is the single most repeated lesson of this project. Try both on your own prompts.

Two things measured while building it. Width beats depth: at equal parameter count, two hidden layers of 8192 reach 0.7691 against 0.7944 for one layer of 16384. And a residual extrapolates worse than a matrix does — outside the corpus it saw, a linear map degrades gracefully while the network collapses.

Measured results

A cosine of 0.79 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts with several distinct cuts and no bleed between them, fl2va with first and last frame, ref2va with a reference image, and since 0.1.3 ref2va with a reference video.

Fidelity does not collapse on short prompts: measured per-token cosine goes from 0.937 at 80 words to 0.908 at 2 words, once the attention sink is handled.

Speech

The first release lost non-English speech: a French line came out half Spanish, and the 8B put everything in English. That was the clearest regression and I could not explain it then.

With mmh3-8b-ClipProj-celeb-mlp, a three-shot clip carrying English, French and Spanish comes out like the 32B does, and the audio level gap measured against the reference has gone from 7.6 dB to 3.5.

Part of what was blamed on the projection was not the projection. A line that fills more than about two thirds of its shot comes out slurred whatever encoder produced the conditioning — the fix is a longer shot, not a better matrix. And MiniMax H3 expects speech wrapped in <d>[Language] ...</d> with a stable speaker id declared beforehand; without that, one voice with one accent is used for the whole clip. Neither of those is documented here because neither is ours, but both cost me a day.

Run the controls first

The two control matrices exist to prove the learned matrix is doing the work rather than the diffusion model. Same prompt, same seed, only the matrix changes:

‖W_identity‖ = 50.6 against ‖W_learned‖ = 52.4 — near-identical energy, so the difference is structural, not a matter of scale.

If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.

What is in obsolete/

The previous matrices, kept because a comparison posted on r/StableDiffusion ran on them and the links have to keep working. They have no name coverage and are calibrated on a corpus thirty times smaller. There is no reason to prefer them.

Among them, the CONDPROJ pair, and the story is worth telling because the mistake was instructive.

The DiT does not consume the conditioning as it arrives: it first passes it through condition_proj, a Linear(5120 → 5376) feeding the token refiner. That layer's spectrum is very uneven — a factor of 45 between the top and bottom deciles of its singular values, 52 % of the energy in 10 % of the directions. Plain ridge regression ignores this and spends as much effort on a direction the DiT will multiply by 0.10 as on one it will multiply by 37. Calibrating against the output of that layer instead, then mapping back through the pseudo-inverse, should therefore minimise the error the DiT actually sees. The cosine went from 0.697 to 0.845 on the 4B and 0.731 to 0.860 on the 8B.

Then I compared what the two matrices actually output:

4B   CONDPROJ against unweighted, same corpus     cosine 0.999998
8B   CONDPROJ against unweighted, same corpus     cosine 0.999999

They are the same function. Unregularised least squares is invariant to an invertible linear transform of the targets, so fitting in one space and mapping back recovers the same map; only the ridge penalty breaks that invariance, and with 37 851 training tokens against λ = 1000 it barely binds. The entire gain was an artefact of measuring in a different space.

The idea came from u/stddealer on r/StableDiffusion, and it was a good one. The measurement is on me: I published the cosine before checking whether the matrix had changed at all.

Known limitations

Quantisation costs facts. Comparing int8_convrot against bf16 on factual recall shows errors appearing under quantisation. Fine for general use, worth knowing if your prompts lean on proper nouns.

Masks defeat identity. A character recognised by a costume rather than a face comes out as an unknown person in the right suit. No corpus fixes that, because the identity is not in the name's representation to begin with.

Counting is unreliable, and not because of the projection. Ask for three of something and you get four, on the 32B too. Enumerating works better than announcing a number.

Required models

The 32B text encoder is no longer needed — that is the entire point.

Licence and responsibility

These matrices are released under MIT, like the node.

They are derived from the activations of both models, and their legal status is unclear. They are provided as-is, for research, with no claim of ownership over anything derived from the underlying models.

• Qwen3-VL is published by Alibaba under Apache 2.0. Read and comply with its terms and acceptable-use policy.
• MiniMax H3 ships under a custom licence. Read it before any use, particularly commercial.

This project is not affiliated with, endorsed by, or connected to Alibaba / Qwen, MiniMax, or Comfy Org.

You remain responsible for what you generate and for complying with the licences of every model you load.

Credits

Vibe-coded with Anthropic Claude Code (Opus 5). Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected. Three claims in the previous version of this file were wrong and are corrected here.