TinyBrainBot 350M V3 Instruct GGUF

This repository provides GGUF quantizations for nkthebass/tinybrainbot-350mV3-instruct.

TinyBrainBot 350M V3 Instruct is a 348M-parameter decoder-only model pretrained from scratch on 22.7B tokens and instruction-tuned across 2,000 SFT steps (131M tokens) with warm-up stable decay (WSD). It answers direct factual questions, summarizes short texts, and provides succinct explanations while fitting in extremely compact compute budgets.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by standard k-quant quantization.

---

Available Files and Quantizations

---

Model Architecture Details

• Parameters: ~348M (348,342,912)
• Architecture: Decoder-only transformer (LlamaForCausalLM)
• Layers: 22
• Hidden Size: 1152
• Attention Heads: 18 query heads / 6 KV heads (GQA), head dimension 64
• Intermediate Size: 3072
• Context Length: 2048 tokens (SFT trained at 1024)
• Vocabulary Size: 32,000 (tbb-32k-v2, tied word embeddings)
• RoPE Theta: 10,000.0

---

Prompt Format and Chat Template

[!IMPORTANT] The TinyBrainBot V3 chat template is space-separated, not newline-separated, with an essential leading space:

<|user|> {prompt} <|end|> <|assistant|>

Constructing prompts with extra newlines degrades generation output.

Example Single-turn Prompt:

 <|user|> What is the boiling point of water at sea level? <|end|> <|assistant|>

Example Multi-turn Prompt:

 <|user|> Hello! Who are you? <|end|> <|assistant|> I am TinyBrainBot, a compact language model. <|end|> <|user|> Can you explain what photosynthesis is? <|end|> <|assistant|>

---

Quickstart Guide

1. llama.cpp

Run interactive instruction generation with llama-cli:

llama-cli -m ./tinybrainbot-350mV3-instruct-Q4_K_M.gguf \
    -p " <|user|> Explain quantum computing in three sentences. <|end|> <|assistant|>" \
    -n 256 \
    --temp 0.7 \
    --top-p 0.9 \
    --repeat-penalty 1.1

Launch a local OpenAI-compatible inference server:

llama-server -m ./tinybrainbot-350mV3-instruct-Q4_K_M.gguf \
    --port 8080 \
    -c 2048

2. Ollama

Create a Modelfile:

FROM ./tinybrainbot-350mV3-instruct-Q4_K_M.gguf
TEMPLATE " <|user|> {{ .Prompt }} <|end|> <|assistant|>"
PARAMETER stop "<|end|>"
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Build and run:

ollama create tinybrainbot-instruct -f Modelfile
ollama run tinybrainbot-instruct "What causes the northern lights?"

3. LM Studio

1. Place the .gguf file in your LM Studio models folder.
2. Select tinybrainbot-350mV3-instruct-Q4_K_M.gguf.
3. Set the prompt template format:
   • System prompt: (Leave empty)
   • User prefix: <|user|>
   • User suffix: <|end|>
   • Assistant prefix: <|assistant|>
   • Stop strings: <|end|>