J-Deutzia-0.5B-SFT-UL

J-Deutzia-0.5B-SFT に unlikelihood 学習を追加した版。チャット形式は「### ユーザー:」/「### アシスタント:」。

469M total / 199M active (4 experts, top-1), 14 layers, ctx 16384, vocab 72k.

import sys, json, torch
from safetensors.torch import load_file
sys.path.insert(0, "code")
from canonical_corpus.probe_moe_lm import MoELMConfig, MoEProbeLM

cfg = {k: v for k, v in json.load(open("config.json")).items() if k not in ("architectures", "model_type")}
model = MoEProbeLM(MoELMConfig(**cfg))
model.load_state_dict(load_file("model.safetensors"), strict=False)

Generation: code/scripts/j72v2_gen_eval.py (generate, ReversibleSentencePieceAdapter).