Gemmuguinho: a compact MoE language model for AAC pictogram boards in Brazilian Portuguese

Gemma + Papuguinho — born to communicate

A sovereign, CPU-friendly Mixture-of-Experts language model that generates Augmentative and Alternative Communication (AAC/CAA) pictogram boards in Brazilian Portuguese — and edits them by natural-language command. Built for Papuguinho to run fully on-device (no GPU) via llama.cpp.

The name Gemmuguinho combines the base architecture (Gemma-3 by Google) with the Papuguinho project it was born from — much like the mascot image: a hatchling with its own identity, grown from another's egg.

TL;DR — Type "Quero uma prancha de escola" → get a ready-to-use 12-pictogram communication board. Ask "adiciona 3 frutas" or "apagar pictograma do cachorro" → get a structured edit plan. ≈212M active params (top-2), ≈90–175 tok/s on 4 CPU threads, no GPU (175 tok/s measured on a 4-vCPU AMD EPYC VM).

• 🧩 Task: AAC pictogram board generation + board editing (speech-therapy / assistive tech)
• 🌎 Language: Brazilian Portuguese (pt-BR)
• ⚙️ Architecture: gemma3moe — Gemma-3-270m upcycled to an 8-expert MoE (top-2)
• 💾 Size: ≈636M total · ≈212M active (k=2, default) · runs on CPU, no GPU needed
• 📦 GGUF builds: gemmuguinho-636m-GGUF

---

Model Description

Gemmuguinho is a small Mixture-of-Experts (MoE) language model specialized for AAC (Augmentative and Alternative Communication). AAC boards are grids of pictograms that non-verbal or minimally-verbal people — often autistic children or people with speech disorders — use to communicate. Building good boards by hand is slow; this model produces coherent, thematically-balanced boards in seconds, on commodity hardware.

It is upcycled from Google's gemma-3-270m: the vocabulary was trimmed to a compact 64k Portuguese-focused set via embedding-vocab-trimmer, and every dense feed-forward block was replaced by a routed MoE block (8 experts, top-2) via gemma3moe-builder. Experts were diversely seeded (Branch-Train-miX style) from gemma-3-270m (base), gemma-3-270m-it (instruct), a board fine-tune, and a distilled variant, then trained on the AAC corpus. The result runs on llama.cpp with a custom gemma3moe architecture.

Intended Use

Two modes

1. Board generation (chatbot): "quero uma prancha de X" → a board (default 12 pictograms; a specific count can be requested, e.g. "gera 5 pictogramas para usar no dentista").
2. Board editing (agent): given the current board + a command, it returns a compact edit DSL — R| confirmation, +| add, -| remove, T| retitle, C| columns, X| new theme.

Each board line is label|category|search_terms (category ∈ s/v/a/e/p/l).

How to Use

llama.cpp (recommended — CPU / on-device)

Use the GGUF repo:

llama-cli -m gemmuguinho-636m-Q8_0.gguf --jinja \
  -sys "$(cat corpo_chatbot.txt)" \
  -p "quero uma prancha de escola" -n 200 -st

Transformers (safetensors, trust_remote_code)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "Papuguinho/gemmuguinho-636m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

msgs = [{"role": "system", "content": SYSTEM_PROMPT},   # see corpo_chatbot.txt
        {"role": "user", "content": "quero uma prancha de escola"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(ids["input_ids"], max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

⚠️ Generate single-sequence (batch size 1). Left-padded batched generation can corrupt outputs due to Gemma-3's sliding-window attention.

Training Data

Brazilian-Portuguese corpus of 44,865 examples (683 held out for validation), mixing:

• Board generation (≈34.7k): themed AAC boards produced by teacher LLMs from a curated pictogram catalog.
• Editing commands (≈10.8k): natural-language edit instructions → edit-DSL plans, including typos and refusal/no-op cases.

Teachers: a mix of Gemini models. No human data; no personal data.

Evaluation

Scored by a discriminative LLM judge (0–10 rubric: theme coherence, concentration/dispersion balance, picturability, usefulness). Reference points on the same rubric: a 1B dense baseline scores 7.8, the teacher ≈ 9.2.

Command-following (edit DSL) is reliable across quantizations, including exact-count adds and id-correct removals.

Limitations & Safety

• pt-BR only; board vocabulary reflects the synthetic training catalog.
• Not a general chatbot and not a source of factual/medical information.
• Built-in scope guard: it only edits children's AAC boards. Health, hygiene, body, and difficult emotions are treated as legitimate; clearly adult or harmful content is refused with a short, gentle message.
• Outputs should be reviewed by a caregiver, educator, or speech-language therapist before use.

Technical Specifications

Related Resources

Citation

@misc{stekel2026gemmuguinho,
  title  = {Gemmuguinho: a compact MoE language model for AAC pictogram boards in Brazilian Portuguese},
  author = {Stekel, Tardelli R. C. and Antonelli, Manuela and Baroni, Gabriel P. and {Carvalho Neto}, José A. and Souza, Marcela J. V.},
  year   = {2026},
  note   = {Papuguinho, https://www.papuguinho.com},
  url    = {https://huggingface.co/Papuguinho/gemmuguinho-636m},
  doi    = {10.57967/hf/9931}
}

Project: Papuguinho · License: Gemma