Yejin Korean 3B v2 SFT

Supervised fine-tuned version of the Yejin Korean 3B v2 base model.

Model Description

This model is a 3-billion parameter Korean language model that has been supervised fine-tuned on high-quality instruction-following datasets.

Base model: Yejin Korean 3B v2
Architecture: Transformer decoder-only
Parameters: ~3B (3072d / 28L / 24H / 8KV)
Vocabulary: 64,000 tokens (custom Korean tokenizer)
Max sequence length: 4,096 tokens
Precision: bfloat16

Training Details

Method: Full parameter fine-tuning (no LoRA/adapters)
Distributed: FSDP across 8x NVIDIA H200 GPUs
Epochs: 3
Learning rate: 2e-5 (cosine schedule)
Batch size: 128 (2 per GPU x 8 GPUs x 8 gradient accumulation)
Weight decay: 0.01
Warmup: 3% of total steps

Training Data

Fine-tuned on curated Korean instruction-following datasets:

Synthetic and instruction data
Multi-turn conversation data
Post-training data
Knowledge-grounded QA data
Agent/tool-use data
Reading comprehension data

Chat Template

<s><|user|>
{instruction}<|end|>
<|assistant|>
{response}<|end|>
</s>

For multi-turn conversations:

<s><|user|>
{user_msg_1}<|end|>
<|assistant|>
{assistant_msg_1}<|end|>
<|user|>
{user_msg_2}<|end|>
<|assistant|>
{assistant_msg_2}<|end|>
</s>

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("cooler8/yejin-korean-3b-v2-sft")
tokenizer = AutoTokenizer.from_pretrained("cooler8/yejin-korean-3b-v2-sft")

messages = [{"role": "user", "content": "한국의 수도는 어디인가요?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

License

Apache 2.0

cooler8/yejin-korean-3b-v2-sft