Nebula (320M) GGUF

This repository provides GGUF quantizations for OrionLLM/Nebula.

Nebula is a 320M-parameter generalist Small Reasoning Model trained on 200B+ tokens, designed for edge AI, embedded systems, and on-device deployment. It is optimized for strong memory, general reasoning, arithmetic, and retrieval-friendly tasks while operating with compact compute and memory overhead.

All GGUF files were converted from the original safetensors weights using llama.cpp at native BF16 precision, then quantized into standard k-quant variants.

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
`Nebula-320M-BF16.gguf`	BF16	614.61 MB	Full precision base conversion. Highest fidelity reference weights.
`Nebula-320M-Q8_0.gguf`	Q8_0	327.74 MB	Near-lossless 8-bit quantization. Recommended for best quality.
`Nebula-320M-Q6_K.gguf`	Q6_K	311.39 MB	High quality retention with minimal degradation. Excellent balance.
`Nebula-320M-Q5_K_M.gguf`	Q5_K_M	244.94 MB	Solid balance between memory usage and generation accuracy.
`Nebula-320M-Q4_K_M.gguf`	Q4_K_M	228.60 MB	Fast, lightweight 4-bit quant. Recommended default for mobile and edge setups.
`Nebula-320M-Q3_K_M.gguf`	Q3_K_M	198.01 MB	Compact footprint when memory headroom is strictly limited.
`Nebula-320M-Q2_K.gguf`	Q2_K	185.35 MB	Maximum compression. Noticeable quality loss, intended for ultra-constrained environments.

Reasoning Format and Stenographic Tokens

Nebula produces reasoning traces with an intentionally compact, stenographic style designed for efficiency under limited parameter capacity. Traces use structured shorthand notations within <think>...</think> blocks:

Logical Markers

Token	Meaning	Usage
`→`	derivation / implication	Short causal and logical flow
`↺`	iterative return / refinement loop	Backtracking, reconsidering priors, RAG re-querying
`?`	uncertainty / questions to resolve	Appended to short expressions and words
`!/※`	insight / breakthrough	Marked knowledge discovery
`≈`	approximation / estimate	Intermediary hypothesis and preliminary statements
`∴`	therefore / final step	Stable conclusion marker

Uncertainty Markers

Token	Meaning	Usage
`●`	high confidence	Well-supported empirical or theoretical anchor points
`◐`	medium / partial confidence	Incomplete data, plausible but unverified links
`○`	low confidence	Speculation, missing context, weak inference chain
`⚠`	bias / premise risk	Domain mismatch, cultural assumptions, language-switch artifacts
`?maybe?`	soft speculation	Tentative branch that may collapse later

Verification Process

Token	Meaning	Usage
`☐`	unverified hypothesis	Raw claim without cross-check
`☑`	intermediate verification	Supported by at least one source or argument
`✓`	confirmed / validated	Multiple independent supports

Prompt Format

Nebula uses ChatML markup with generation prompts opening into <think> blocks:

<|im_start|>system
You are a helpful reasoning assistant.<|im_end|>
<|im_start|>user
Calculate the square root of 1764 and explain your steps.<|im_end|>
<|im_start|>assistant
<think>

How to Use

1. With llama.cpp

Run interactive generation with llama-cli:

llama-cli -m ./Nebula-320M-Q4_K_M.gguf \
    -p "<|im_start|>system\nYou are a helpful reasoning assistant.<|im_end|>\n<|im_start|>user\nWhat is the capital of France and why is it culturally significant?<|im_end|>\n<|im_start|>assistant\n<think>\n" \
    -n 512 \
    -c 4096 \
    --temp 0.6

Launch a local OpenAI-compatible API server:

llama-server -m ./Nebula-320M-Q4_K_M.gguf \
    --host 127.0.0.1 \
    --port 8080 \
    -c 4096

2. With Ollama

Create a file named Modelfile in the same directory:

FROM ./Nebula-320M-Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
<think>
"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|end_of_text|>"
PARAMETER temperature 0.6

Build and run the model:

ollama create nebula-320m -f Modelfile
ollama run nebula-320m

3. With LM Studio

Place the desired .gguf file into your local LM Studio models directory.
Select Nebula-320M in the model loader tab.
Use the ChatML preset as the prompt template.

Model Architecture Details

Original weights: OrionLLM/Nebula
Architecture: LlamaForCausalLM (model_type: llama)
Parameters: ~320M
Layers: 80
Hidden size: 576
Intermediate size: 1536
Attention heads: 9 query heads, 3 key/value heads (GQA)
Head dimension: 64
Context window: 4,096 tokens
Vocabulary size: 65,536 tokens
Native precision: bfloat16
License: Apache 2.0

Attribution & Credits

Original model training and architecture by OrionLLM.

GGUF conversions produced by Pink Pixel.

Made with 💖 by Pink Pixel

sizzlebop/Nebula-320M-GGUF