Qwen3.5-4B-Base sparse autoencoders: 65K features, mean L0 128

Two SAE dictionaries trained on Qwen3.5-4B-Base activations, each for exactly 500,000,000 training-vector exposures. The model and processor revision is 1001bb4d826a52d1f399e183466143f4da7b741b.

The dictionary size is 65,536. Training uses BatchTopK and low-frequency AuxK; the middle SAE gives image and text groups separate mean-L0=128 budgets. This is an average budget, not exactly 128 nonzeros in every token. The visual SAE additionally uses invertible principal-direction conditioning and training-frequency bias adjustment. Training exposures may reuse cached vectors.

Evaluation

fixed is the trained feedforward encoder with frozen calibrated thresholds. pursuit uses the frozen learned decoder dictionary and nonnegative residual sparse pursuit, with each row's calibrated fixed-mode L0 as its budget. Pursuit is the default, costs more compute, and can select features absent from the feedforward support. A pursuit coverage result does not mean the trained feedforward encoder has zero inactive features.

The visual test bank has 1M patches; the middle test bank has 500K image and 500K text positions. Dead10M refers only to the training main branch.

Both pursuit modes pass the project's numerical reconstruction/coverage targets. Fixed visual test inactivity exceeds the <1% target; fixed middle global FVU exceeds the <=0.10 target. Raw visual FVU alone is insufficient because the first principal direction explains about 99.896% of input variance. Orthogonal FVU removes this direction. Metrics cannot rank different hooks directly.

In pursuit mode, 65,536 visual and 65,530 middle features have support from at least ten independent test image groups. This does not certify semantic purity or causal usefulness. Reconstruction replacement on 128 test image groups gives middle joint-image-and-text delta-CE +0.01480 (95% CI [0.01073, 0.01786]). Training stopped at the 500M budget, with no additional convergence plateau test. Full metrics and uncertainty estimates are in evaluation.json and each dictionary's final_test.json.

Download and use

No base-model weights are included. These are custom SAE modules, loaded with the supplied PyTorch code.

pip install huggingface_hub
hf download AwesomeInterpretability/qwen3.5-4b-base-saes-65k-k128 --local-dir ./qwen_saes
pip install -r ./qwen_saes/requirements.txt

To download only one dictionary:

from huggingface_hub import snapshot_download
snapshot_download(
    "AwesomeInterpretability/qwen3.5-4b-base-saes-65k-k128",
    local_dir="qwen_saes",
    allow_patterns=["qwen_visual/*", "load_sae.py", "requirements.txt", "README.md"],
)

Replace qwen_visual with qwen_middle for the other dictionary.

import sys
import torch
from pathlib import Path
sys.path.insert(0, str(Path("qwen_saes").resolve()))
from load_sae import load_sae
torch.backends.cuda.matmul.allow_tf32 = False

visual = load_sae("qwen_saes", "qwen_visual", device="cuda")
# hidden: [N, 1024], valid raw hook activations BEFORE normalization.
# Process small batches; pursuit is evaluated with batches of 64.
# ids, values = visual.encode_sparse(hidden[:64])
# reconstructed = visual.reconstruct(hidden[:64])
# fixed_codes = visual.encode(hidden[:64], mode="fixed")

middle = load_sae("qwen_saes", "qwen_middle", device="cuda")
# hidden: [N, 2560]; roles: [N] integer tensor, 1=image, 2..5=text.
# ids, values = middle.encode_sparse(hidden[:64], roles[:64])
# reconstructed = middle.reconstruct(hidden[:64], roles[:64])

The loader applies saved mean/scale normalization and, for visual pursuit, conditioning automatically. Do not normalize the inputs again. The middle roles are 1=IMAGE, 2=QUESTION, 3=ANSWER, 4=CAPTION, 5=CONTEXT_AFTER_IMAGE. Exclude padding, template tokens, special markers, and ineligible text spans. Capture the full block output; if a hook returns a tuple, select its hidden tensor.

encode_sparse returns candidate feature IDs and coefficients; keep values

1e-7 for support statistics. encode returns a dense [N, 65536] tensor despite the sparse contents; large N can exhaust memory. CPU loading is supported; CUDA is recommended for pursuit. Use the supplied reconstruct to map back to original hook coordinates. Float32 matmul with TF32 disabled was used for final pursuit evaluation.

Training curves and provenance

pip install tensorboard
tensorboard --logdir ./qwen_saes/training --port 6006

Open Custom Scalars for the core metrics. The x axis is cumulative training vectors. Learning rate warms up over 5M vectors, stays at 3e-4 until 400M, then cosine-decays to 3e-5 at 500M. Each training/<dictionary>/ directory contains selected-run lineage, complete logged scalar history, validation snapshots, and TensorBoard events. Earlier visual branches are identified in the lineage; their exposures are not double counted in the selected trajectory.

Each dictionary includes safetensors weights, frozen thresholds, normalization, portable configuration, inference code, and per-feature support/decoder-neighbor statistics. Visual requires both raw-space weights and conditioned_core plus conditioning for the two inference modes. Geometry statistics for the visual dictionary are in the conditioned training coordinates. The source-artifact identifiers in JSON describe provenance, not additional downloadable paths.

Training data provenance

The activation-source identifiers are cc3m, gqa, obelics, localized_narratives, and pixmo_cap. The per-modality training budgets are recorded in each dictionary's budget.json. This repository distributes SAE parameters and aggregate statistics; it does not distribute the source images, captions, questions, answers, or activation banks. Source-artifact references are provenance identifiers, not paths that a user needs to mount.

License

The SAE weights and supplied inference code are released under the Apache License, Version 2.0; see LICENSE. The separately downloaded base model, Qwen/Qwen3.5-4B-Base, also uses Apache-2.0. The base-model weights are not included in this repository.