Yejin Korean 8B v3 cpt2s20000 + Qwen3.6-27B distill mixK3.1 r128 soup5 nothink-v2
Korean instruction-following language model.
Chat Template
<s><|user|>\n{instruction}<|end|>\n<|assistant|>\n<think> </think>{response}<|end|>
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("cooler8/yejin-korean-8b-v3-cpt2s20000-distill-qwen36-mixK31-r128-soup5-nothink-v2", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("cooler8/yejin-korean-8b-v3-cpt2s20000-distill-qwen36-mixK31-r128-soup5-nothink-v2")
messages = [{"role": "user", "content": "한국의 수도는 어디인가요?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
inputs.pop("token_type_ids", None)
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Training
Five-part model soup (weighted weight average: 2x seed 0 + 1x seed 7 + 1x seed 11 + 1x seed 13) of four rank-128 (alpha 256, lr 2e-4) LoRA fine-tunes of the same Yejin model: Yejin 8B v3 (pretrained from scratch by Yejin: cooler8/yejin-korean-8b-v3-base) after two stages of continued pretraining on public Korean data (6,000 steps in stage 1 plus 20,000 steps in stage 2, which includes FinePDFs Korean text partly extracted by an OCR model, RolmOCR), with the Yejin chat vector from our own SFT model. Each LoRA member was trained in a distinct data shuffle order (seeds 0, 7, 11, 13) on mixK3.1 (90,275 samples), which removes single-line shortcut templates and applies an Answer-First + Deep Chain-of-Thought + Answer-Confirm format across knowledge, reading comprehension, and multi-step narrative reasoning items (including short Korean solutions generated by Qwen/Qwen3.6-27B under Apache-2.0 for questions from our own puzzle generator, KMMLU train, AI Hub school exams, AI Hub knowledge and reading comprehension; only solutions reaching the known answer were kept via sequence-level distillation with no teacher weights or logits). No benchmark test items were used, and the training data share zero word 13-grams with KMMLU-Pro, KMMLU test, CLIcK, MuSR or Korean MuSR items.
License
Apache 2.0