BananaMind-1.5-Base

BananaMind-1.5-Base is a small English causal language model trained from scratch by BananaMind.

It is our first fully pretrained medium model

Model Details

A instruction tuned version is coming very soon.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "BananaMind/BananaMind-1.5-Base"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=False)

model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=False,
    dtype=dtype,
).to(device)

model.eval()

prompt = "The color of the sky is blue. The color of a banana is"
inputs = tok(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=16,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tok.eos_token_id,
        eos_token_id=tok.eos_token_id,
    )

print(tok.decode(out[0], skip_special_tokens=True))

Generation Settings

Recommended starting settings:

temperature = 0.7
top_p = 0.9
max_new_tokens = 64

For deterministic sanity tests:

do_sample = False
max_new_tokens = 8

Training

BananaMind-1.5-Base was trained from scratch on approximately 27B tokens of FineWeb-Edu-style English web text.

The model uses a custom 32k byte-level BPE tokenizer and a compact Llama-style architecture with grouped-query attention.

Architecture

BananaMind-1.5-Base uses a compact Llama-style decoder architecture:

• 12 Transformer layers
• 640 hidden size
• 1728 intermediate size
• 10 attention heads
• 5 key-value heads
• grouped-query attention
• SiLU activation
• RMSNorm
• tied input/output embeddings
• 4096 token context length

Evaluation

Our model performs very good in comparison to other models:

Parameter vs Size

Citation

@misc{bananamind15base,
  title = {BananaMind-1.5-Base},
  author = {BananaMind},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/BananaMind/BananaMind-1.5-Base}}
}