Korean | English
v1.2.1, ๋ผ๋ฒจ ํ๋กํ์ผ edu-v1+fmt ยท 29ํด๋์ค BIO 22์ธต ๊ต์ฌ ๋ชจ๋ธ)mulkzz/kestrel-edu-ner-22l(v1.2.1, ๋ผ๋ฒจ ํ๋กํ์ผ edu-v1+fmt)๋ ๊ฒฝ๋ ์๋น ํ์ ๋ชจ๋ธ(mulkzz/kestrel-edu-ner-6l)์ ๋ค์ค ์ ํธ ์ง์ ์ฆ๋ฅ(Stage 2)๋ฅผ ๊ฐ๋
ํ๊ธฐ ์ํด ํ์ต๋ 22์ธต ์์ ๊ต์ฌ(Teacher) ํ ํฐ ๋ถ๋ฅ ๋ชจ๋ธ(149.34M ํ๋ผ๋ฏธํฐ / ๋น์๋ฒ ๋ฉ 110.94M)์
๋๋ค. skt/A.X-Encoder-base(9708f9c, Apache-2.0) ์์ 4๋ ํต์ฌ ๋น์ ํ ๊ฐ์ฒด(PERSON, SCHOOL, ORG, ADDRESS) + 10์ข
์ ํ ๋ณด์กฐ ๊ฐ์ฒด(PHONE, EMAIL, RRN, PASSPORT, DRIVER_LICENSE, ACCOUNT, CARD, BRN, URL, CODE) ์ด 29ํด๋์ค BIO ํค๋๋ฅผ ์น๊ณ , ์กฐ์ฌ ๋ถ๋ฆฌ ์ ๋ ฌ(particle_aware)๊ณผ ๋ํ์ด์์ด(amb ๋ amb_neg) + ๋๋ช
์ด์ธ(D1=B ๊ต๊ณผ ๊ณต์ธ O ๋ ์ค์ ํ์ PERSON) ๋์ผ ๋ฏธ๋๋ฐฐ์น ๋์กฐ ์ํ๋ฌ๋ก ๋ฏธ์ธ์กฐ์ ํ์ต๋๋ค.
69.62 MB INT8 ONNX, CPU 2.17ร ๊ฐ์, dorms-pii-ner-bench v1.1 post-v1.1 ์ ์ค๋ฅ 91.6% [589/643] / ๋๋ฝ 8.4%).main ยท v1.2.1 ยท v1.2: 29ํด๋์ค edu-v1+fmt ๊ต์ฌ ๊ฐ์ค์น(sha256: 9998e608...) ๋ฐ v1.2.1 ์ต์ ๋ฒค์น๋งํฌ ๋ํ๋ฅผ ์ ๊ณตํฉ๋๋ค. ์ด๊ธฐ 9ํด๋์ค(edu-v1) ๋จ๋ฐ ๊ธฐ์ค์ ์ revision="v1.0" (133302653e95c30d0e8ea0729f0e6f9649d4a0eb)์ ์๊ตฌ ๋ณด์กด๋ฉ๋๋ค.C-4(4 vCPU) ๊ธฐ์ค 20,000์ร๋์ 4์์ฒญ ํฌ์ ์ง์ฐ์ด ~54.5s(40s ์ํ ์ด๊ณผ)์ด๋ฏ๋ก ์ค์๊ฐ ๊ฒ์ดํธ์จ์ดยท์จ๋๋ฐ์ด์ค ๋ธ๋ฆฌ์ง์๋ 6์ธต ์ฆ๋ฅ ํ์ ๋ชจ๋ธ์ ์ฌ์ฉํ๋ฉฐ, ๋ณธ ์ ์ฅ์๋ ์ง์ ์ฆ๋ฅ ์ฌํยท์คํ๋ผ์ธ ๊ณ ์ ๋ฐ ๊ธฐ์ค์ ยทํ์ ์ฐ๊ตฌ์ฉ์ผ๋ก ๊ณต๊ฐํฉ๋๋ค.0๊ฑด, ์ธ๋ถ ์์ฉ LLM ์์ฑ ํ์ต ๋ฐ์ดํฐ 0๊ฑด(28,000๊ฑด ๊ฒฐ์ ๋ก ์ ํฉ์ฑ ๊ต์ก ๋ฌธ์).post-v1.1 ๋ฒค์น๋งํฌ ๋ฐ ๊ต์ฌ-ํ์ ์ฉ๋ ๋น๊ตFig. 1 | 22์ธต ๊ต์ฌ(
kestrel-edu-ner-22l) โ 6์ธต ํ์(kestrel-edu-ner-6l) ์์ถ ์ ์ง์ฐยท๋ฉ๋ชจ๋ฆฌ ์ ๊ฐ ๋ฐv1.2.14์ค ๊ฒ์ดํธ ์์ฝ. ๋น์๋ฒ ๋ฉ ์ฐ์ฐ ํ๋ผ๋ฏธํฐ๋ฅผ110.94M โ 30.70M(-72.3%)์ผ๋ก ์ค์ฌ Local Bridge 20k์ ์ง์ฐ์6.49s โ 1.82s, PII Worker20kร4์ง์ฐ์17.03s โ 4.40s๋ก ๋จ์ถํ๋ฉด์๋ ๋ฏธ๋ ธ์ถ ๋จ๋ฐitest_v1_2(3,000๊ฑด) ๋งคํฌ๋ก ํต์ฌ ์ฌํ์จ97.84% โ 97.44%(์ ํ 1,817 span100.0% โ 100.0%)๋ฅผ ์ ์งํจ. 6์ธต ํ์์dorms-pii-ner-bench v1.1(post-v1.1, ๋๋ฝ ๋ฌธํญ17.0% โ 8.4%, $p < 0.0001$) ์์ธ ์ฑ์ ์ mulkzz/kestrel-edu-ner-6l ์นด๋ ๋ฐ./figures/ner_bench_v1_1_6l.svg์ฐธ์กฐ.
Fig. 2 | Stage 1 22์ธต ๊ต์ฌ ๋ฏธ์ธ์กฐ์ ์๋ ด ๋์ญํ ๋ฐ ๋ถํฌ ๋ด(
dev_v1_2/ ๋จ๋ฐitest_v1_23,000๊ฑด) ์ฑ์ .28,000๊ฑด ํฉ์ฑ ๊ต์ก ๋ฌธ์์์ ์ฐจ๋ฑ ํ์ต๋ฅ (backbone 3e-5,classifier 1e-4, ์ฝ์ฌ์ธ ์ค์ผ์ค, ์คํจ ๋ฐฐ์น32,bf16)๋ก ํ์ตํ์ฌ1.5์ํญ(step=1,783,7.44Mํ ํฐ)์์ ์ต๊ณ ๊ฒฐํฉ ์ ์(selection_score=1.7843, ์ ํ ์ฌํ์จ100.0%, ๊ณต์ธ ์คํ2.31%)๋ฅผ ๋ฌ์ฑํจ.
train 28,000 / dev_v1_2 3,000 / ๋จ๋ฐ itest_v1_2 3,000). ์ค์ ํ์ ๋ฐ์ดํฐ 0๊ฑด, ์ธ๋ถ ์์ฉ LLM ์์ฑ๋ฌธ 0๊ฑด.AdamW (backbone_lr=3e-5, classifier_lr=1e-4, weight_decay=0.01, warmup_ratio=0.06, max_grad_norm=1.0, micro_batch=16 ร grad_accum=2), ์ฌ๋กฏ ๋์ ์นํ ์ฆ๊ฐ 15%, ๋ค์ค ์ค์ผ์ผ ์ฐฝ(W-400 30% ํผํฉ).ํ๊ฐ ์์ (epoch) |
๊ธ๋ก๋ฒ ์คํ | ๋์ ์ ํจ ํ ํฐ | dev_loss |
selection_score |
macro_core_recall |
formal_recall |
homonym_student |
public_person_fp |
๋น๊ณ |
|---|---|---|---|---|---|---|---|---|---|
| 0.5 | 593 | 2.55M | 0.00408 | 1.7837 | 98.58% | 100.00% | 97.57% | 0.33% | ์ด๊ธฐ ์๋ ด |
| 1.0 | 1,187 | 4.95M | 0.00446 | 1.7700 | 98.55% | 100.00% | 93.69% | 0.33% | no_imp=1 |
| 1.5 | 1,783 | 7.44M | 0.00620 | 1.7843 | 98.60% | 100.00% | 97.09% | 2.31% | ์ต์ข
์ฑํ (*BEST*) |
| 2.0 | 2,379 | 9.91M | 0.00496 | 1.7584 | 98.62% | 100.00% | 96.60% | 8.91% | no_imp=1 |
| 2.5 | 2,976 | 12.40M | 0.00861 | 1.7708 | 98.64% | 100.00% | 96.12% | 8.25% | no_imp=2 |
| 3.0 | 3,573 | 14.88M | 0.01840 | 1.7785 | 98.66% | 100.00% | 98.06% | 9.24% | ์กฐ๊ธฐ ์ข
๋ฃ (patience=3) |
dev_v1_2 (3,000๊ฑด) ๋ฐ ๋ฏธ๋
ธ์ถ ๋จ๋ฐ itest_v1_2 (3,000๊ฑด) ๋น๊ต (W-400, pipeline ๋ทฐ)| ๋ถํ | ๋ชจ๋ธ | macro_core_recall |
overall_core_recall |
formal_recall |
PERSON |
SCHOOL |
ORG |
ADDRESS |
public_person_fp (NER) |
|---|---|---|---|---|---|---|---|---|---|
dev_v1_2 (3,000) |
kestrel-edu-ner-22l (๊ต์ฌ) |
98.71% | 98.32% | 100.00% | 98.24% | 100.00% | 96.60% | 100.00% | 2.31% |
dev_v1_2 (3,000) |
kestrel-edu-ner-6l (๋ณด์ ํ์) |
98.45% | 97.84% | 100.00% | 97.69% | 99.60% | 96.60% | 99.90% | 5.28% |
itest_v1_2 (3,000) |
kestrel-edu-ner-22l (๊ต์ฌ) |
97.84% | 96.79% | 100.00% | 96.60% | 100.00% | 94.90% | 99.85% | 0.00% |
itest_v1_2 (3,000) |
kestrel-edu-ner-6l (๋ณด์ ํ์) |
97.44% | 95.77% | 100.00% | 95.44% | 99.56% | 94.75% | 100.00% | 3.74% |
transformers) ๋ฐ ๊ณ ์งimport torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
REPO_ID = "mulkzz/kestrel-edu-ner-22l"
REV = "v1.2.1" # ๋๋ ๊ธฐ๋ณธ ๋ธ๋์น "main"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision=REV)
model = AutoModelForTokenClassification.from_pretrained(REPO_ID, revision=REV).eval()
text = "๋ณ๋น๊ณ ๋ฐ ๋ฏธ๋ ํ์๊ณผ ์ด๋กฑ์ด๊ฐ ํด์ค๋ฆ์๋ด์ผํฐ(๋ฌผ์๊ฐ๋ก 58, 1์ธต)์ ๋ฐฉ๋ฌธํ๋ค."
inputs = tokenizer(text, return_offsets_mapping=True, return_tensors="pt")
offsets = inputs.pop("offset_mapping")[0].tolist()
with torch.no_grad():
pred = model(**inputs).logits[0].argmax(dim=-1).tolist()
for pid, (s, e) in zip(pred, offsets):
if e > s and pid != 0:
print(model.config.id2label[pid], text[s:e], (s, e))
pad_token_id = 49999): ์๋ ๋ฐฐ์น ํจ๋ฉ ์ ๋ฐ๋์ 49999๋ฅผ ์ฌ์ฉํ์ญ์์ค(0์ [CLS]).townboy/korean-pii-dataset CC BY 4.0, KDPII Zenodo 10968609 CC BY 4.0, ์ฝ๊ธฐ ์ ์ฉ ์ง๊ณ๋ง ์ฌ์ฉ) ์์ธ ๊ฒฐ๊ณผ๋ mulkzz/kestrel-edu-ner-6l ๋ชจ๋ธ ์นด๋๋ฅผ ์ฐธ์กฐํ์ญ์์ค. ๋ผ์ด์ ์ค๋ Apache-2.0 (NOTICE)์
๋๋ค.skt/A.X-Encoder-base