ABA-1 (124M) — Czech and English Conversational Code Model

ABA-1 is a decoder-only Transformer trained from scratch. It is intended for Czech and English conversation and programming fundamentals across Python, C, C#, JavaScript, HTML, and CSS.

Model details

Training snapshot

• Pretraining: 3,000 steps on an 8.12 million-token Czech, English, and programming corpus. Recorded validation loss: 1.0367.
• Chat fine-tuning: checkpoint aba_ai_chat_best.pt, step 500, validation loss 0.8775.
• The local dataset snapshot contained 4,554 dialogues (4,098 training and 456 validation).
• Hardware used for training: NVIDIA GeForce RTX 3050 Laptop GPU with 4 GB VRAM.

Use with a Transformers pipeline

Install the runtime packages:

pip install torch "transformers>=4.50.0" tokenizers safetensors

The model implementation is custom code, so trust_remote_code=True is required when loading the model. The tokenizer uses the standard PreTrainedTokenizerFast wrapper over the included tokenizer.json.

from transformers import pipeline

repo_id = "SeightCZE/ABA1-1.1.0-0.12B"
generator = pipeline(
    "text-generation",
    model=repo_id,
    tokenizer=repo_id,
    trust_remote_code=True,
)

prompt = "<|user|>\nHello, who are you?\n<|assistant|>\n"
result = generator(
    prompt,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.25,
    top_p=0.9,
)
print(result[0]["generated_text"])

Use with AutoTokenizer and AutoModelForCausalLM

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "SeightCZE/ABA1-1.1.0-0.12B"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

prompt = "<|user|>\nWrite a Python function that checks whether a score is above 50.\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.25,
        top_p=0.9,
    )
new_tokens = output_ids[0, inputs["input_ids"].shape[1] :]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The model uses a 512-token sliding context window and does not use a KV cache. Generation is compatible with the standard Transformers generate() API, but will be slower than architectures with cached attention.

Run the included chat program

On Windows, install the dependencies with install_hf.bat, then start run_chat_hf.bat. The launcher loads the exported model through AutoModelForCausalLM and the tokenizer through AutoTokenizer.

Files

• model.safetensors — exported model weights.
• config.json — model configuration and AutoClass mapping.
• generation_config.json — recommended generation defaults.
• configuration_aba.py, modeling_aba.py — custom Transformers model implementation.
• tokenizer.json, tokenizer_config.json — tokenizer data and Transformers metadata.
• chat_hf.py, run_chat_hf.bat, install_hf.bat — standalone chat and setup scripts.
• tokenizer.json, tokenizer_config.json — serialized fast tokenizer and Transformers metadata; AutoTokenizer loads the tokenizer from tokenizer.json.
• vocab.json, merges.txt — standalone BPE vocabulary and merge rules matching the data bundled in tokenizer.json (16,384 tokens and 16,116 merge rules).
• LICENSE — project license.

License

MIT License. See LICENSE.