Type-o1-mini-instruct - GGUF

GGUF quantizations of kd13/Type-o1-mini-instruct, a compact general-purpose instruct model (~1B parameters) for everyday assistant use.

Converted with llama.cpp. The IQ quant was produced with an importance matrix; the rest are static quants.

Read the Usage section before running these files. This model uses a custom chat template, so llama.cpp requires the --jinja flag. Without it you will get this custom template is not supported.

Provided quants

Sorted by size, which is not the same as sorted by quality. IQ-quants are often preferable to non-IQ quants of a similar size.

Which one should I pick?

For a model this small the practical range is Q4_K_M through Q8_0. A 1B model has little redundancy to give up, so the very low-bit quants lose more than they would on a 7B. Q2_K and Q3_K_S are included for completeness rather than as recommendations.

Chat template

This model was fine-tuned on a custom template, not the standard Llama 3 header format. Each message is wrapped as:

''' <|begin_of_text|>{role} {content}<|end_of_text|> '''

and generation is prompted with a trailing <|begin_of_text|>assistant\n. The full Jinja template is embedded in every GGUF file in this repo, so any runtime with Jinja support applies it automatically.

Because this format is not one of llama.cpp's built-in recognised templates, its C++ template matcher will reject it. Passing --jinja tells llama.cpp to use the embedded Jinja template instead, which is what you want.

Usage

llama.cpp

llama-completion -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja \
  -sys "You are a helpful assistant." \
  -p "Explain photosynthesis in two sentences."

Recent llama.cpp builds renamed llama-cli to llama-completion; on older builds use llama-cli with the same flags. For raw text completion with no template applied at all, add -no-cnv and drop --jinja.

Server:

llama-server -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja -c 4096

Omitting --jinja produces this custom template is not supported — that is a template-matching error, not a corrupt file.

Ollama

ollama run hf.co/kd13/Type-o1-mini-instruct-GGUF:Q4_K_M