Qwen3-0.6B contextual chunk

A 0.6B model that writes one or two sentences placing a document chunk inside its parent document. It fills the contextual retrieval step of a RAG ingest pipeline: the index stores each chunk together with the context it came from, which helps retrieval when a chunk on its own is ambiguous. The model is small enough to run at ingest time on the machine that later answers queries, so document text never leaves the device.

Domain: Russian contracts, procurement and technical documentation.

Files

The merged weights and the adapter are the same network. Use the merged weights unless you want to keep the adapter separate from the base model.

Quality

Heuristic word-F1 against the teacher reference, measured on the same 60 held-out validation rows before and after fine-tuning:

Per category, base and fine-tuned:

word-F1 clears 0.10 on all 60 rows after training, against 47 of 60 for the base model. The metric compares the output with the teacher's own wording, so on its own it says little about retrieval quality. The end-to-end numbers are in the technical report linked below.

Training

The training data pairs 3,134 chunks with the document window each was cut from and the context an API teacher wrote for it (dataset card).

• base model Qwen/Qwen3-0.6B, loaded in 4-bit, LoRA rank 16 with alpha 16 attached to every linear layer, 10.09M trainable parameters (1.665% of the model)
• 2,701 train and 433 validation rows, split by window_hash so that identical prompts never straddle the two splits
• response-only loss masking: the document window and the instruction are masked out and the loss covers the assistant turn alone
• two epochs, effective batch size 8 (batch 2 with 4 gradient accumulation steps), learning rate 2e-4 with 5 warm-up steps, maximum sequence length 16,384; the longest training sample is 9.7k tokens, so nothing is truncated
• 676 steps on a single T4, about 9.5 hours, resumed across four Colab sessions; the best checkpoint by evaluation loss is step 640
• training and inference use the same prompt, a byte-identical Qwen3 chat template with thinking disabled

Usage

With the merged weights:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "p4ulbr4dl3y/qwen3-0.6b-contextual-chunks"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16, device_map="auto").eval()

TEMPLATE = """<document>
{document}
</document>
Вот фрагмент, который нужно соотнести со всем документом:
<chunk>
{chunk}
</chunk>
Сформулируй краткий контекст (1-2 предложения), чтобы привязать этот фрагмент к документу в целом для улучшения поисковой выдачи. Ответь только кратким контекстом и ничем больше."""

msgs = [{"role": "user", "content": TEMPLATE.format(document=document, chunk=chunk)}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=160, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True).strip())

With the adapter, if you would rather keep the base weights separate:

from peft import PeftModel

model = PeftModel.from_pretrained(base_model, REPO, subfolder="lora").eval()

The prompt has to reach the model exactly as written above, wrapped in the Qwen3 chat template with enable_thinking=False. A mistake in the assembly shows up as a different first token rather than as an obviously broken answer.

GGUF

gguf/qwen3-0.6b-contextual-chunks-Q8_0.gguf runs the same network in llama.cpp, Ollama or LM Studio, which covers CPUs and non-Apple hardware. The ablation used the same 60 validation rows; the bfloat16 reference on those rows is 0.537.

Q8_0 is the default here. Its difference from f16 is not significant, and no quantization produced an empty answer or a number missing from the document. Q6_K is a fair pick under a tight size budget, while Q4_K_M loses quality measurably at 0.6B. Only Q8_0 is uploaded; the other builds take seconds with the conversion scripts in the repository. The intervals are a paired bootstrap over the 60 rows.

llama-cli -m gguf/qwen3-0.6b-contextual-chunks-Q8_0.gguf -ngl 99 -c 16384 -p "<chat-templated prompt>"

Limitations

• The output is a retrieval aid, not a summary, so it compresses and paraphrases rather than reproduces the document faithfully.
• Russian contracts, procurement and technical documentation only. Quality on other domains is unmeasured.
• word-F1 scores agreement with the teacher's wording, which is a proxy for retrieval quality and not a measurement of it.
• The final model put no invented numbers into its 60 validation answers. The check only covers numbers that also fail to appear in the document window, so it will not catch invented entities, and an intermediate epoch-1 checkpoint did insert two numbers that were absent from the source.

Links

• Technical report and retrieval harness: https://github.com/p4ulbr4dl3y/contextual-chunks
• Training data: https://huggingface.co/datasets/p4ulbr4dl3y/ru-contextual-chunks