BananaMind-2-Mini

BananaMind-2-Mini is a small decoder-only causal language model trained from scratch by BananaMind on a 30B-token curriculum. It is our first model in the BananaMind 2 Series!

The model has 25,178,752 parameters, a 4,096 token context window, and a custom 8k-token digit-aware byte-level BPE tokenizer.

Model Details

Credits to AxiomicLabs and GPT X2 125M for the architecture inspiration.

Tokenizer

BananaMind-2-Mini uses a custom 8k byte-level BPE tokenizer trained from FineWeb-Edu text with digit-aware pre-tokenization.

Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.

Digit IDs:

Examples:

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Training Data

BananaMind-2-Mini was trained on a 30B-token mix of web, educational, synthetic textbook, and math data.

The run used a progressive curriculum rather than sampling the final aggregate mix from the first token.

Training Setup

Evaluation

Self-reported benchmark scores using lm_eval. Scores may vary a bit depending on harness version, runtime settings, dtype, and evaluation environment.

All task scores below use acc_norm,none. Average is the mean over ARC Easy, PIQA, ARC Challenge, and HellaSwag.

Higher is better on the y-axis, and more parameters are farther right on the x-axis. The top-left region represents the most score-efficient models.

Repository Files

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "BananaMind/BananaMind-2-Mini"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Suggested Generation Settings

For stable continuations:

• do_sample=False
• repetition_penalty=1.1
• max_new_tokens=64 to 160

For more varied text:

• do_sample=True
• temperature=0.6 to 0.8
• top_p=0.9
• top_k=50
• repetition_penalty=1.1
• max_new_tokens=64 to 192

Intended Use

BananaMind-2-Mini is intended for lightweight language-model research, local experimentation, text continuation, tokenizer experiments, and small-model training comparisons.

Because this is a base model, prompts should be written as continuation prompts rather than chat messages.

License

Apache 2.0

🍌