berturk-punctuation-restoration

Fine-tuned dbmdz/bert-base-turkish-cased (BERTurk base cased) with a token classification head for Turkish punctuation and casing restoration. No existing punctuation checkpoint was used; only the pretrained BERTurk encoder weights and its original tokenizer.

Task formulation

• Input: whitespace-split words, Turkish-lowercased, without ,.;:!?.
• One label per word, predicted on the word's last subword token; other subwords and [CLS]/[SEP]/[PAD] use ignore index -100.
• Label = CASE|PUNCT: casing of the word and the punctuation mark that follows it.
• Words are never inserted, deleted, reordered or respelled; only casing and the trailing mark change.
• Inputs longer than 64 subword tokens are split into overlapping word windows (12 words overlap); each word takes the prediction from the window where it has the most context.

Training data

GoktugD/turkish-punctuation-restoration-500k @ fcbb30d342b511fdd8bf1f09bbfef990443752fa (synthetic, template-based; official splits).

Preprocessing: NFC; removal of the stray U+0307 produced by locale-unaware str.lower('İ') in the dataset sources; model inputs are re-derived from the targets with Turkish-aware lowercasing (I→ı, İ→i) and verified word-by-word against the dataset sources (0 alignment errors). Validation/test use first-sentence templates and entities disjoint from train; the second sentence of every row comes from 8 templates shared across splits (QUESTION and EXCLAMATION occur only there).

Training procedure

Evaluation

Word-level metrics. Punctuation macro F1 excludes O.

Long input = consecutive rows concatenated into one input and processed with overlapping windows.

Per-class punctuation (test)

Confusion matrix (test, rows = gold)

First sentence (split-specific templates) vs later sentences (shared templates), test

Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

repo = "thealper2/berturk-punctuation-restoration"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()
PUNCT = {"O": "", "COMMA": ",", "PERIOD": ".", "QUESTION": "?", "EXCLAMATION": "!", "COLON": ":", "SEMICOLON": ";"}

def tr_lower(s): return s.replace("I", "ı").replace("İ", "i").lower()
def tr_upper(s): return s.replace("i", "İ").replace("ı", "I").upper()

def restore(text: str) -> str:
    words = [tr_lower(w.strip(",.;:!?")) for w in text.split() if w.strip(",.;:!?")]
    enc = tokenizer(words, is_split_into_words=True, truncation=True, max_length=64, return_tensors="pt")
    with torch.no_grad():
        pred = model(**enc).logits.argmax(-1)[0].tolist()
    labels = {}
    for pos, wid in enumerate(enc.word_ids()):
        if wid is not None:
            labels[wid] = model.config.id2label[pred[pos]]  # last subword of each word wins
    out = []
    for i, w in enumerate(words):
        case, _, punct = labels.get(i, model.config.id2label[0]).rpartition("|")  # case is "" without casing labels
        if case == "CAP":
            w = tr_upper(w[:1]) + w[1:]
        elif case == "UPPER":
            w = tr_upper(w)
        out.append(w + PUNCT[punct])
    return " ".join(out)

print(restore("bugün hava çok güzel dışarı çıkabiliriz"))

Single window only; for longer texts split into overlapping word windows.

Limitations

• Trained and evaluated only on synthetic, template-based text (12 first-sentence and 8 second-sentence templates); scores measure in-distribution performance, not performance on natural or spoken Turkish.
• Only the marks present in the training data are supported: ,, ., ?, !.
• No evaluation on ASR (e.g. Whisper) output; disfluencies, repetitions, recognition errors and digit formatting are unseen.
• Casing of unseen proper nouns is inferred from context only.