Qwen3.5-4B-text-only

Text-only language model extracted from Qwen/Qwen3.5-4B, a Qwen3.5 multimodal (image+video+text) model.

What was removed

• The vision encoder (Qwen3_5VisionModel, ~SigLIP-style ViT used to embed images/video frames)
• The vision patch merger / projector that maps vision features into the language model's embedding space
• The image/video token routing logic in Qwen3_5Model.forward (pixel_values, image_grid_thw, etc.)
• The auxiliary multi-token-prediction (mtp.*) head present in the original checkpoint (already ignored at load time by the original model too - see _keys_to_ignore_on_load_unexpected in the upstream code)

What was kept

• Token embeddings (model.embed_tokens)
• The full hybrid decoder stack: gated linear attention (Qwen3_5GatedDeltaNet) layers interleaved with standard full-attention layers every full_attention_interval layers, exactly as in the original text backbone (model.language_model in the original checkpoint -> model here)
• Final norm (model.norm) and LM head (lm_head)
• Tokenizer (unchanged - plain text tokenizer, no image/video preprocessor)

Parity verification

Outputs were checked against the original multimodal model on text-only prompts (no images/video). Greedy generate() decodes were verified to match token-for-token, and logits matched with a max absolute difference of 0.00e+00 (bf16 numerical noise floor).

Loading

This architecture (qwen3_5_text / Qwen3_5ForCausalLM) is natively supported in transformers>=5.9. No trust_remote_code is required:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("tuandunghcmut/Qwen3.5-4B-text-only", dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("tuandunghcmut/Qwen3.5-4B-text-only")

inputs = tok("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=30)
print(tok.decode(out[0], skip_special_tokens=True))

A standalone reference implementation (modeling_qwen3_5_text.py + configuration_qwen3_5_text.py, verified bit-for-bit identical to the native transformers implementation on random weights) is included in this repo for transparency/portability. It is not required for loading - transformers already ships this architecture natively - but documents exactly what the text-only forward pass does.

Original model

See Qwen/Qwen3.5-4B for the full multimodal model, license, and training details. This repo inherits the apache-2.0 license from the base model.