flame-27m-instruct GGUF

This repository provides GGUF quantizations for ifx-pse-sys-ml/flame-27m-instruct.

flame-27m-instruct is a compact 27.1M-parameter English instruction-tuned model trained on FineWeb-Edu-dedup, Cosmopedia-v2, ClimbMix, and FineMath, fine-tuned on SmolTalk. At ~5x smaller than SmolLM-135M-Instruct, it serves as an ultra-lightweight decoder for edge devices, embedded environments, and rapid local experimentation.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by quantization into standard k-quant variants.

---

Available Files and Quantizations

---

Architecture Specifications

• Parameters: 27.1M
• Layers: 8
• Hidden size: 512
• Attention heads: 8 (Query) / 2 (Key/Value) with Grouped Query Attention (GQA)
• Head dimension: 64
• Intermediate size: 1280 (SwiGLU)
• Query/Key Normalization: Per-head RMSNorm on Q and K projections
• Context length: 2048 tokens
• RoPE base frequency: 1,000,000 (NeoX style)
• Vocabulary: 12,000 (English ByteLevel BPE)
• Embeddings: Tied word embeddings (tie_word_embeddings: true)

---

Benchmarks

Accuracy (%) evaluated via lm-evaluation-harness 0.4:

---

Prompt Format

flame-27m-instruct uses standard ChatML markup:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Tell me about the moon in one sentence.<|im_end|>
<|im_start|>assistant

---

How to Use

1. With llama.cpp

Run interactive generation with llama-cli:

llama-cli -m ./flame-27m-instruct-Q4_K_M.gguf \
    -p "<|im_start|>user\nTell me about the moon in one sentence.<|im_end|>\n<|im_start|>assistant\n" \
    -n 64 --temp 0.3

Launch a local HTTP server:

llama-server -m ./flame-27m-instruct-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 -c 2048

2. With Ollama

Create a Modelfile:

FROM ./flame-27m-instruct-Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.3

Then create and run:

ollama create flame-27m -f Modelfile
ollama run flame-27m

3. With LM Studio

1. Copy flame-27m-instruct-Q4_K_M.gguf into your LM Studio models directory.
2. Select ChatML as the prompt preset.
3. Load the model and begin testing.