Chinese Classical GPT-2

A 335M parameter GPT-2 model trained from scratch for style-conditioned classical Chinese text generation, with post-training for Li Bai persona emulation.

Overview

This project implements a complete pipeline from pre-training to persona-based dialogue:

1. Pre-training: Two-stage curriculum learning (general Chinese → classical Chinese) with Style Embedding for 5 literary genres
2. Post-training: Continual Pre-training (CPT) + Supervised Fine-Tuning (SFT) for Li Bai persona

Model Variants

This repository contains multiple checkpoints from different training stages and ablation experiments:

Architecture

Style Personas (Pre-training)

Training

Stage 1 — General Chinese Pre-training

• Data: 1.68M samples (classical + modern Chinese)
• Result: Loss 10.36 → 4.0, Accuracy 2.5% → 33.5%

Stage 2 — Classical Chinese Specialization

• Data: 1.60M samples (classical Chinese only, with style labels)
• Result: Loss 4.0 → 3.85, Perplexity 42.43

Post-training — Li Bai Persona (CPT + SFT)

• CPT: 1,329 Li Bai texts (poems, prose, biographies), Loss 4.30 → 1.34
• SFT: 1,000 multi-turn dialogues in Li Bai's voice, Loss 3.76 → 0.58
• Hardware: NVIDIA RTX 4080 SUPER (16GB), ~10 min total

Evaluation

LLM-Judge Quality (Tasks 1-5, scored 0-100)

Adversarial Robustness (Task 6, scored 0-100)

Persona Identification (open-ended, by DeepSeek judge)

Repository Structure

├── checkpoints/
│   ├── stage1_final.pt          # Pre-training Stage 1
│   └── stage2_final.pt          # Pre-training Stage 2 (GPT2-Raw)
├── checkpoints_post/
│   ├── cpt_final.pt             # Post-training CPT (with SE)
│   └── sft_final.pt             # Post-training SFT (GPT2-SE, primary)
├── tokenizer/
│   ├── chinese_sp.model          # SentencePiece BPE tokenizer
│   └── chinese_sp.vocab
└── evaluation/
    ├── questions.json            # 130 evaluation questions
    ├── results_posttrain_style.json
    ├── results_posttrain_nostyle_fair.json
    ├── results_posttrain_nostyle_unfair.json
    └── results_baseline.json

Usage

Persona Dialogue (Post-trained model)

import torch
import sentencepiece as spm
from model import GPT2
from config import ProjectConfig, STYLE_ID_MAP

sp = spm.SentencePieceProcessor()
sp.load("tokenizer/chinese_sp.model")

config = ProjectConfig()
config.model.vocab_size = sp.get_piece_size()
model = GPT2(config.model, pad_token_id=0)
state = torch.load("checkpoints_post/sft_final.pt", map_location="cpu", weights_only=True)
model.load_state_dict(state)
model.eval()

prompt = "[STYLE:李白]\n### 系统：你是大唐诗人李白\n### 用户：写一首思乡的诗\n### 回答："
input_ids = [sp.bos_id()] + sp.encode(prompt)
idx = torch.tensor([input_ids])

output = model.generate(
    idx, max_new_tokens=256, style_id=STYLE_ID_MAP["李白"],
    temperature=0.7, top_k=20, top_p=0.8, repetition_penalty=1.3,
)
print(sp.decode(output[0, len(input_ids):].tolist()))

Limitations

• Local coherence only: 335M parameters cannot maintain long-range narrative logic
• Style bleeding: Style signal attenuates in longer outputs (>200 tokens)
• Potential SFT overfitting: Low SFT loss (0.58) on 1,000 examples × 10 epochs
• No explicit prosodic supervision: Tonal patterns learned incidentally through statistical co-occurrence

Citation

@misc{chinese-classical-gpt-2026,
  title={Cross-Era Alignment for Emulating Ancient Chinese Literati},
  author={Zichao Wei and Entang Wang and Zhenyu Feng},
  year={2026},
  howpublished={Software Project Neural Networks, Saarland University}
}

License

MIT