Fine-Tuned POS Tagging Model - SK_Morph_BLM (POS Tags)

Model Overview

This model is a fine-tuned version of the SK_Morph_BLM model for tokenization and POS tagging. For this task, we used the UD Slovak SNK dataset, which is part of the Universal Dependencies project. This dataset contains annotated Slovak texts with various linguistic information, including UPOS tags, morphological features, syntactic relations, and lemmatization. We focused on UPOS tags, which provide basic categories of parts of speech.

POS Tags

Each token in the dataset is annotated with one of the following POS tags:

• NOUN (0): Nouns
• PUNCT (1): Punctuation marks
• VERB (2): Verbs
• ADJ (3): Adjectives
• ADP (4): Adpositions (Prepositions)
• PRON (5): Pronouns
• PROPN (6): Proper nouns
• ADV (7): Adverbs
• DET (8): Determiners
• AUX (9): Auxiliary verbs
• CCONJ (10): Coordinating conjunctions
• PART (11): Particles
• SCONJ (12): Subordinating conjunctions
• NUM (13): Numerals

Unused tags:

• X
• INTJ
• SYM

Dataset Details

The UD Slovak SNK dataset contains annotated Slovak texts that we adapted for this task, fine-tuning the model for POS tagging. The dataset provides UPOS tags for each token, which allowed us to refine our model for accurate recognition and categorization of parts of speech in the Slovak language. The total number of sequences in the data set we used is 9,847.

Fine-Tuning Hyperparameters

The following hyperparameters were used during the fine-tuning process:

• Learning Rate: 3e-05
• Training Batch Size: 64
• Evaluation Batch Size: 64
• Seed: 42
• Optimizer: Adam (default)
• Number of Epochs: 10

Model Performance

The model was evaluated using stratified 10-fold cross-validation, achieving a weighted F1-score with a median value of 0.982.

Model Usage

This model is suitable for tokenization and POS tagging of Slovak text. It is specifically designed for applications requiring accurate categorization of parts of speech in various texts.

Example Usage

Below is an example of how to use the fine-tuned SK_Morph_BLM-pos model in a Python script:

import torch
from transformers import RobertaForTokenClassification
from huggingface_hub import hf_hub_download, snapshot_download
import json

class TokenClassifier:
    def __init__(self, model, tokenizer):
        self.model = RobertaForTokenClassification.from_pretrained(model, num_labels=14)
        
        repo_path = snapshot_download(repo_id = tokenizer)
        sys.path.append(repo_path)

        # Import the custom tokenizer from the downloaded repository
        from SKMT_lib_v2.SKMT_BPE import SKMorfoTokenizer
        self.tokenizer = SKMorfoTokenizer()
        
        # Stiahnutie a načítanie JSON súboru s mapovaním
        byte_utf8_mapping_path = hf_hub_download(repo_id=tokenizer, filename="byte_utf8_mapping.json")
        with open(byte_utf8_mapping_path, "r", encoding="utf-8") as f:
            self.byte_utf8_mapping = json.load(f)

    def decode(self, tokens):
        decoded_tokens = []
        for token in tokens:
            for k, v in self.byte_utf8_mapping.items():
                if k in token:
                    token = token.replace(k, v)
                token = token.replace("Ġ"," ")
            decoded_tokens.append(token)
        return decoded_tokens

    def tokenize_text(self, text):
        encoded_text = self.tokenizer.tokenize(text.lower(), max_length=256, return_tensors='pt', return_subword=False)
        return encoded_text

    def classify_tokens(self, text):
        encoded_text = self.tokenize_text(text)
        tokens = self.tokenizer.convert_list_ids_to_tokens(encoded_text['input_ids'].squeeze().tolist())
        
        with torch.no_grad():
            output = self.model(**encoded_text)
            logits = output.logits
            predictions = torch.argmax(logits, dim=-1)
            
            # Použitie masky založenej na attention mask
            active_loss = encoded_text['attention_mask'].view(-1) == 1
            active_logits = logits.view(-1, self.model.config.num_labels)[active_loss]
            active_predictions = predictions.view(-1)[active_loss]

            probabilities = torch.softmax(active_logits, dim=-1)
            
            results = []
            for token, pred, prob in zip(self.decode(tokens), active_predictions.tolist(), probabilities.tolist()):
                if token not in ['<s>', '</s>', '<pad>']:
                    result = f"Token: {token: <10}  POS tag: ({self.model.config.id2label[pred]} = {max(prob):.4f})"
                    results.append(result)
                    
        return results

# Inštancia POS token klasifikátora s použitím špecifikovaného tokenizéra a modelu
classifier = TokenClassifier(tokenizer="daviddrzik/SK_Morph_BLM", model="daviddrzik/SK_Morph_BLM-pos")

# Text na klasifikáciu
text_to_classify = "Od učenia ešte nikto nezomrel, ale načo riskovať."

# Klasifikácia tokenov tokenizovaného textu
classification_results = classifier.classify_tokens(text_to_classify)
print(f"============= POS Token Classification =============")
print("Text to classify:", text_to_classify)
for classification_result in classification_results:
    print(classification_result)

Example Output Here is the output when running the above example:

============= POS Token Classification =============
Text to classify: Od učenia ešte nikto nezomrel, ale načo riskovať.
Token:  od         POS tag: (ADP = 0.9976)
Token:  učenia     POS tag: (NOUN = 0.9891)
Token:  ešte       POS tag: (PART = 0.9775)
Token:  nikto      POS tag: (PRON = 0.8371)
Token:  nezomr     POS tag: (VERB = 0.9961)
Token: el          POS tag: (VERB = 0.9917)
Token: ,           POS tag: (PUNCT = 0.9990)
Token:  ale        POS tag: (CCONJ = 0.9914)
Token:  načo       POS tag: (ADV = 0.9188)
Token:  riskovať   POS tag: (VERB = 0.9955)
Token: .           POS tag: (PUNCT = 0.9991)