qwrt/Swedish0.1M

This is a byte-level language model based on the qwen3 architecture, which means it predicts letter instead of word chunks+. It consists of about 133k parameters and was trained on about 7 billion total characters of Swedish text. I have tested the models ability to predict the next letter of a sequence by analysing the percentage of correctly predicted next letter from the story "rödluvan och vargen"(https://godnattsagan.se/sv/rodlovan-och-vargen). The model sees at most 110 characters back and it predicts the next character 59% of times. The training took about 2 hours on a rtx 4070 laptop with a speed of 1300k/s. By examining model outputs, I have estimated a model vocab of over 3000 swedish words(see ordförråd.txt, ranked by frequency)

Since this is a relatively small llm, it is not so good at producing coherent text.

from transformers import Qwen3ForCausalLM
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Qwen3ForCausalLM.from_pretrained("qwrt/Swedish0.1M").to(device)

seed = torch.tensor(list("Hej på di".encode()), dtype=torch.long)[None].to(device)
out = model.generate(seed, max_new_tokens=128,
                                     do_sample=True, temperature=0.8)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

output:

Hej på dig. Med en sekund var inte hon bara att hon hade kunnat svara på hög med att avslå med den röda döden? 
Den skulle Lisa att 

If you want to generate more than 128 tokens, run this code, it downloads sink_generate.

from huggingface_hub import hf_hub_download
import importlib.util
from transformers import Qwen3ForCausalLM
import torch

repo="qwrt/Swedish0.1M"

file_path = hf_hub_download(
    repo_id=repo,
    filename="sink_generate.py"
)

spec = importlib.util.spec_from_file_location("sink_generate", file_path)
modul = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modul)


device = "cuda" if torch.cuda.is_available() else "cpu"
model = Qwen3ForCausalLM.from_pretrained(repo).to(device)

seed = torch.tensor(list("Det var en gång".encode()), dtype=torch.long)[None].to(device)
out = modul.sink_generate(model, seed, max_new_tokens=2000, window=120, n_sink=0)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

By examining 100 000 samples with 1024 character each with the model, it is possible to give an estimate of the model total vocabulary of swedish words. To know if a word is generated by the LM is real or not, we compare it to a large vocabulary list. This vocabulary list was made by extracting all words from the training set. A reasonable estimate for the models vocabulary size would be around 5000 words. The following graph shows how the amount of "real" words diminishes when analysing words used by the model, ordered from most frequent to less. frequent. For example, 99.9% of the top 4 500 words used by the model are real words, and 99% of all top 7 500 words used by the model are real.