---

Qwen3-VL-4.3B-semantic

An expanded-vocabulary variant of Qwen/Qwen3-VL-4B-Instruct, designed for use as a text encoder in Krea 2.

The original 36 decoder layers and vision tower are retained. The main architectural modification is an expanded embed_tokens / tied lm_head vocabulary containing additional whole-word English tokens.

What changed

Existing token IDs are preserved. The added vocabulary consists of English words that were previously represented by more than one tokenizer subword.

The additional embedding rows contain semantic initialization rather than random initialization.

Semantic initialization

The added embeddings were constructed in several stages:

1. English lexical candidates were selected from the external lexical resources described below.
2. Initial semantic representations were obtained from ConceptNet Numberbatch.
3. WordNet-derived hypernym relationships were used to impose additional lexical structure.
4. The semantic representations were aligned to the original Qwen embedding space using a linear least-squares projection based on the mean embeddings of the corresponding original subword representations.
5. The resulting vectors were used to initialize the added Qwen vocabulary rows.

This procedure should be understood as semantic embedding expansion and initialization, not as full pretraining or continued pretraining of the Qwen3-VL transformer.

Krea 2 compatibility

This model is intended to replace the original Qwen3-VL-4B text encoder and tokenizer in a Krea 2-compatible pipeline.

Krea 2 uses intermediate Qwen3-VL hidden states from layers:

2, 5, 8, 11, 14, 17,
20, 23, 26, 29, 32, 35

with hidden size 2560.

The resulting representation is expected to have the form:

(batch, sequence_length, 12, 2560)

Use the tokenizer distributed with this repository. The original Qwen3-VL tokenizer does not contain the additional vocabulary and therefore cannot emit the new token IDs.

Usage

from transformers import AutoTokenizer, Qwen3VLForConditionalGeneration

repo = "Sorenvza/Qwen3-VL-4_3B-semantic"

tokenizer = AutoTokenizer.from_pretrained(repo)

model = Qwen3VLForConditionalGeneration.from_pretrained(
    repo,
    torch_dtype="auto"
)

This repository is intended to provide the modified Qwen3-VL text-conditioning component. A compatible Krea 2 implementation is required for image generation.

Data and external resources

This model incorporates information derived from several external lexical and semantic resources.

ConceptNet Numberbatch

ConceptNet Numberbatch 19.08 was used as the initial semantic representation for the added vocabulary.

Source:

• ConceptNet
• ConceptNet Numberbatch
• Version: 19.08

The applicable ConceptNet / Numberbatch license and attribution requirements apply to the corresponding derived data.

WordNet / Open Multilingual WordNet

WordNet-derived lexical relations, including hypernym information, were used during semantic embedding construction.

The local lexical environment used the NLTK omw-1.4 resource where applicable.

The applicable WordNet / OMW licensing and attribution requirements apply to the corresponding derived data.

Wiktionary / Wiktextract

Wiktionary-derived lexical information was obtained through Wiktextract / Kaikki where applicable.

The exact dump or snapshot used to construct the released lexical data should be recorded for reproducibility.

Wiktionary-derived data is subject to the applicable Wiktionary licensing terms, including CC BY-SA and GFDL where applicable.

Reproducibility

The following should be recorded for each release:

• Base model revision
• Tokenizer revision
• Lexical-resource versions
• Dataset/download dates
• Semantic vocabulary construction script
• Number of accepted words
• SHA-256 checksums of source resources
• Semantic initialization configuration
• Projection configuration

The released vocabulary contains 116,654 additional word tokens.

Limitations

The added vocabulary does not by itself constitute additional language-model training.

In particular, adding a semantic embedding for a word does not guarantee that the transformer has learned new factual knowledge about that word or that Krea 2 will consistently render every added concept.

The primary purpose of this release is to provide an experimental expanded lexical/semantic interface for downstream conditioning experiments.

License

The Apache-2.0 declaration applies to the released model artifacts to the extent permitted by the licenses of the underlying model and external resources.

Users are responsible for complying with the licenses and attribution requirements of:

• Qwen3-VL
• ConceptNet / Numberbatch
• WordNet / OMW
• Wiktionary-derived resources
• Any other third-party resources used to construct or process the vocabulary