Model Card for SmolLM-135M-de

A german version of HuggingFaceTB/SmolLM-135M, trained to speak German by applying CPT for about 6 billion tokens.

If you are looking for a chat model, try this fine tune or the corresponding adapter model.

Model Details

Model Description

The base model is HuggingFaceTB/SmolLM-135M, which I further trained on about 6 billion German-language tokens.

• Model type: Large Language Model (Llama architecture)
• Language(s) (NLP): German
• License: Apache 2.0
• Finetuned from model: HuggingFaceTB/SmolLM-135M

Uses

I mainly made this as a small experimentation model to quickly benchmark datasets etc. - since the model is so small, I am unsure about its usefulness for any real-world scenarios.

This is a base model without any chat fine tuning etc. and thus should not be used as-is. It outputs mostly correct German, which is what I tried to achieve.

If you are looking for a chat model, try this adapter.

Bias, Risks, and Limitations

This is a very small model and will output blatantly wrong information. I have not done any further filtering on the source datasets, so it is possible that the model will generate lewd or otherwise inappropriate content. Use with care.

I would strongly recommend against using this model in a production setting, at least without further fine tuning and preference optimization.

How to Get Started with the Model

Use the code below to get started with the model.

# adapted from the original SmolLM repo
# pip install transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "LemiSt/SmolLM-135M-de"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
inputs = tokenizer.encode("Rezept für einen leckeren veganen Schokokuchen:\n", return_tensors="pt").to(device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

Training Details

Training Data

• devngho/culturax-mini-nonshuffled
• maxidl/FineNews-unfiltered CC-NEWS-2024-05 config, de split
• djstrong/oscar-small unshuffled_deduplicated_de config
• LemiSt/gutenberg_de
• almanach/HALvest de config
• wikimedia/wikipedia 20231101.de config
• D4ve-R/terra-xplain-cc-de

Training Procedure

This was trained with axolotl, using full fine tuning (no LoRA etc). I used a sequence length of 2048 with an effective batch size of 512, learning rate of 0.003 with the adamw_bnb_8bit optimizer and a cosine scheduler. Due to an error I made in calculating the token count, I accidentally trained for nearly 2 epochs, with the learning rate not reaching its proper minimum.