Raon-1 mini preview: calibrated FP8 KV-cache scales

This derivative preserves the original MXFP8 expert weights and E8M0 weight scales from KRAFTON/Raon-1-mini-preview at revision 7bf87c45fc7b658d3c9532305f3aaed106bbac4f. It adds 14 FP32 scalar parameters for a plain FP8 E4M3FN MLA KV cache, two scalars for each of seven MLA layers. The seven original weight shards are byte-identical to the pinned source.

Calibration

Calibration completed on all 512 selected UltraChat train_sft samples, totaling 437,514 tokens. Samples came from source rows 0 through 511 at dataset revision 8049631c405ae6576f93f445c6b8166f76f5505a, shuffled with seed 42 and truncated to 2,048 tokens using the source tokenizer and chat template. No thinking override was used.

Each observer measures the normalized 512-channel compressed latent plus the raw 64-channel positional tail. One scale per MLA layer is the corpus absolute maximum divided by 448, the E4M3FN maximum. K and V scalar entries share that scale because the native MLA cache stores one compressed latent. No RDA recurrent-state tensors or expanded attention K/V tensors were calibrated. Model weights were not requantized.

Dataset sample IDs, order, revisions, input hash, template hash, observer evidence, and calibration code are included. The data preparation protocol follows LLM Compressor's FP8 KV example. Only its data protocol was used, not its weight quantization recipe.

Runtime requirements and validation limits

This is an offline-calibrated artifact. Full GPU serving, generation quality, cached-prefix reuse, and chunked-prefill GPU execution were not tested. No serving performance claim is made for this derivative.

The scale target is the native Raon MLA implementation in krafton-ai/vllm at revision 9886842e20102e330389a505debe80d4c834940a, with the included required runtime patch. Stock runtime compatibility is not established.

The patch adds Raon-local scalar loading/finalization and restores FP8 cached-prefix latent values before projection and attention. Use the plain E4M3 FP8 cache path. The scales were not calibrated for E5M2, fp8_ds_mla, or NVFP4 KV caches. config.json retains quant_method: mxfp8 and adds kv_cache_quant_algo: FP8. This metadata does not select a serving cache dtype by itself.

The derivative Transformers class registers the 14 scalar parameters for checkpoint readback. Its custom forward does not implement the targeted FP8 MLA cache. Loading these parameters in Transformers alone does not enable FP8 KV execution.

Checks passed for all 19,434 original tensor keys, shapes, and dtypes; seven unchanged weight shards; 14 finite positive FP32 scales; a 19,448-key derivative state dictionary; and CPU scalar loading, finalization, absorption, and inverse-scale computation. Focused CPU tests cover nonunit cached-prefix scale restoration. Each observer saw all 512 samples and 437,514 rows. Saved error metrics apply to each observer's maximum-magnitude row, not an aggregate quality evaluation.

Calibration used PyTorch 2.11.0+cu130, Transformers 5.13.1, safetensors 0.8.0, and LLM Compressor 0.13.0. The upstream card below specifies its own runtime versions; it describes the original model, not validation of this derivative.

License and upstream documentation

Apache License 2.0. The original license, substantive model card, published results, and release history are retained below. Upstream benchmark results are not new measurements of this calibrated derivative.

---

Original upstream model card

Raon-1 mini

Raon-1 mini is a 30B-A4B Mixture-of-Experts (MoE) language model, designed and trained from scratch by KRAFTON AI. It demonstrates strong capabilities in Korean conversation and agentic tasks. As part of the Raon open model family, it is released under the Apache 2.0 license.

[!TIP] Want to try Raon-1 mini without setting anything up?
Chat with it for free, for a limited time, in the Raon-1 mini playground.

[!NOTE] This is a preview release, not a final model. Specs and results may change before the final version.

Highlights

• Reasoning: Built-in thinking, with leading results on the math and science benchmarks shown below.
• Instruction following and search: Leads the comparison models on the instruction-following and search benchmarks shown below.
• Multilingual capabilities: Trained on Korean, English, Japanese, Chinese, Spanish, French, and Thai. Demonstrates strong Korean performance and solid English capabilities.
• Agent execution: Tool calling for agentic workflows, with support for Hermes Agent, Pi and OpenCode.
• Speculative decoding support: A DFlash2 draft model is provided for accelerated speculative decoding.
• Quantization options: Available in MXFP8 and NVFP4 formats, with NVFP4 enabling inference on a single NVIDIA RTX 5090 gpu (32 GB VRAM).

Model Overview

Raon-1 mini uses a hybrid Mixture-of-Experts (MoE) architecture, consisting of 18 linear attention layers and 7 full attention layers.

• Raon Delta Attention (RDA): Raon Delta Attention decouples the erase and write gates of Kimi Delta Attention, which improves long-context performance over KDA.
• Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek’s manifold-constrained hyper-connections (mHC), which speeds up training.
• Hybrid layout: 18 of the 25 layers are Raon Delta Attention layers with a fixed-size state. Only the 7 MLA layers keep a KV cache, about 8 KiB per token in BF16.
• Long context: Natively supports up to 262,144 tokens of context.
• Sparse MoE: 6 of 128 routed experts plus 1 shared expert are active per token, so each token uses 4.3B of the 28.8B total parameters.

---

Benchmark Results

• Red indicates the best performance of Raon-1 mini preview among MoE baselines.

Judge and user simulator: gpt-5.6-luna for all evaluations.
Red = Raon-1 mini preview is best within the MoE group (Raon-1 mini preview, Qwen3.6-35B-A3B, Gemma 4 26B-A4B). Dense models (Qwen3.8-27B, Muse Glimmer 30B) are shown for reference only.
¹ SWE-Verified: evaluated with the mini-SWE-agent harness.
² GDPVal (dataset): metric is a judge score, not Elo.
³ AutomationBench: public split, not the official evaluation set.
⁴ GDP.pdf: official Artificial Analysis (artificialanalysis.ai) text-only version. PDFs are OCR-ed in advance; no image input.
⁵ Chatbot Korean (internal): an internal benchmark measuring Korean colloquial ability in chatbot conversations across diverse situations. Not publicly released.

Sampling parameters

[!Important] Raon-1 mini is designed for thinking mode; disabling it is not recommended.

Settings shared by all tasks:

Settings that depend on the task:

Speculative Decoding

We release a DFlash2 draft model alongside Raon-1 mini preview to accelerate inference through speculative decoding. In our benchmarks on a single NVIDIA B300, it delivers up to 2.18× higher output throughput than autoregressive decoding. Speedups vary by hardware and workload.

Throughput is measured in output tok/s, with speedup over autoregressive decoding in parentheses. Acceptance length (AL) is the mean number of tokens per verification step, averaged across requests at concurrency 1.

AR: autoregressive decoding; C: concurrency. Throughput is total output tokens divided by end-to-end wall time.
Evaluated with vLLM 0.28.0 and an MXFP8 target model; block size 8 (7 draft tokens), probabilistic draft sampling, temperature 1.0, top-p 0.95, top-k 20, thinking enabled, up to 4,096 new tokens, and 128 prompts per benchmark (80 for MT-Bench).

Quickstart

Raon-1 mini supports inference with Transformers, vLLM, llama.cpp, and MLX. The examples below show how to run the model locally or launch an OpenAI-compatible server, using thinking mode and the recommended sampling settings above.

The model repositories are gated. Request access once on this page (approval is automatic; it covers every Raon-1 mini preview repository), then log in so the examples below can download them:

pip install -U huggingface_hub
hf auth login

Transformers

Use Hugging Face Transformers to load Raon-1 mini and generate responses directly in Python.

Install the dependencies:

uv venv --python 3.12 --seed
source .venv/bin/activate

pip install \
    "transformers==5.16.1" \
    "accelerate==1.15.0"

Then run the model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "KRAFTON/Raon-1-mini-preview"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Tell me about KRAFTON AI."}
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    enable_thinking=True,
    truncate_history_thinking=False,
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=32768,
    do_sample=True,
    temperature=1.0,
    top_p=0.95,
    top_k=20,
    min_p=0.0,
    repetition_penalty=1.0,
    presence_penalty=1.5,
)

# The reasoning ends at </think>; the answer follows it.
response = outputs[0][inputs["input_ids"].shape[-1]:].tolist()
think_end = tokenizer.convert_tokens_to_ids("</think>")
split = response.index(think_end) + 1 if think_end in response else 0
reasoning = tokenizer.decode(response[:split], skip_special_tokens=True).strip()
answer = tokenizer.decode(response[split:], skip_special_tokens=True).strip()
print("Reasoning:\n" + reasoning)
print("\nAnswer:\n" + answer)

presence_penalty is applied by the model's remote code, since Transformers has no built-in presence penalty. For coding and math, set presence_penalty to 0.0.

On a 32 GB GPU, use the base MXFP8 model with device_map="auto" and CPU offload.

vLLM (beta)

Use vLLM to serve Raon-1 mini through an OpenAI-compatible API with tool calling and DFlash2 speculative decoding. Tested on NVIDIA B300. The RTX 5090 NVFP4 profile is documented below.

Install from our fork and start the server:

VLLM_REF="raon-1-mini-preview"

uv venv --python 3.12
source .venv/bin/activate

VLLM_USE_PRECOMPILED=1 uv pip install \
    "vllm @ git+https://github.com/krafton-ai/vllm.git@${VLLM_REF}"

vllm serve KRAFTON/Raon-1-mini-preview \
    --host 127.0.0.1 \
    --port 8000 \
    --served-model-name Raon-1-mini-preview \
    --max-model-len 262144 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8_e4m3 \
    --enable-prefix-caching \
    --spec-method dflash \
    --spec-model KRAFTON/Raon-1-mini-preview-dflash2 \
    --spec-tokens 7 \
    --reasoning-parser raon_1 \
    --enable-auto-tool-choice \
    --tool-call-parser raon_1 \
    --default-chat-template-kwargs '{
      "enable_thinking": true,
      "truncate_history_thinking": false
    }'

The first start compiles the model's kernels and takes about 10 minutes; later starts reuse them. To serve the NVFP4 checkpoint, replace the model with KRAFTON/Raon-1-mini-preview-nvfp4 in the same command. It runs on a single GPU (tensor parallel size 1).

RTX 5090 (32 GB)

Use the NVFP4 checkpoint with the following settings. CUDA Toolkit 13 with nvcc on PATH is required.

# WSL2
export VLLM_WSL2_ENABLE_PIN_MEMORY=1

vllm serve KRAFTON/Raon-1-mini-preview-nvfp4 \
    --host 127.0.0.1 --port 8000 \
    --served-model-name Raon-1-mini-preview \
    --max-model-len 32768 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 8 \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8_e4m3 \
    --kv-cache-memory 3221225472 \
    --enable-prefix-caching --async-scheduling \
    --spec-method dflash \
    --spec-model KRAFTON/Raon-1-mini-preview-dflash2 \
    --spec-tokens 7 \
    --reasoning-parser raon_1 \
    --enable-auto-tool-choice --tool-call-parser raon_1 \
    --default-chat-template-kwargs '{"enable_thinking":true,"truncate_history_thinking":false}'

DGX Spark (GB10)

Use the NVFP4 checkpoint with the ARM64 container below. It includes the Raon vLLM fork and the SM 12.1 compatibility fixes for DGX Spark. Docker with NVIDIA GPU support is required. This configuration was tested on GB10 with NVIDIA driver 580.159.03.

After accepting model access and running hf auth login above, download the tested target and draft revisions:

hf download KRAFTON/Raon-1-mini-preview-nvfp4 \
    --revision 0b0f06ac2b3d03abea62dd221fe7839fddf36dfe
hf download KRAFTON/Raon-1-mini-preview-dflash2 \
    --revision 6450d1546d17ee1bb1cb942f316c7fb91cdd868a

Start the OpenAI-compatible server with DFlash2 enabled:

docker run --rm --name raon-spark-quickstart \
    --hostname raon-spark --gpus all --ipc=host \
    -p 127.0.0.1:8000:8000 \
    -v "$HOME/.cache/huggingface:/hf" \
    -v raon-spark-cache:/cache \
    -e HF_HOME=/hf -e HF_HUB_OFFLINE=1 \
    -e MODEL_ID=/hf/hub/models--KRAFTON--Raon-1-mini-preview-nvfp4/snapshots/0b0f06ac2b3d03abea62dd221fe7839fddf36dfe \
    -e DRAFT_MODEL_ID=/hf/hub/models--KRAFTON--Raon-1-mini-preview-dflash2/snapshots/6450d1546d17ee1bb1cb942f316c7fb91cdd868a \
    -e SPEC_TOKENS=7 \
    -e MAX_MODEL_LEN=131072 \
    ghcr.io/krafton-ai/raon-vllm-spark:sm121-4e09534

The image is public; no GHCR login is required. Model access still uses the Hugging Face download step. If you use a custom Hugging Face cache, replace the host cache path in the volume mount.

The first launch compiles CUDA kernels. Keep the raon-spark-cache volume and hostname to reuse them on later launches. Wait until curl -f http://localhost:8000/health succeeds, then send the request below. The served model name is Raon-1-mini-preview.

This profile uses a 128K context window, at most eight sequences, a 2,048-token scheduler batch, and a 3 GiB FP8 KV cache. Concurrent long requests may queue. Set SPEC_TOKENS=0 for autoregressive decoding.

Container source and build instructions. Image digest: sha256:c116b968d899b98c3941a82ad328fbf4a0a80ce7e5034cc0b21a1aa3d8537f88.

Send a request from another terminal:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Raon-1-mini-preview",
      "messages": [
        {
          "role": "user",
          "content": "Tell me about KRAFTON AI."
        }
      ],
      "max_tokens": 8192,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 20,
      "min_p": 0.0,
      "repetition_penalty": 1.0,
      "frequency_penalty": 0.0,
      "presence_penalty": 1.5,
      "chat_template_kwargs": {
        "enable_thinking": true,
        "truncate_history_thinking": false
      }
    }'

Reasoning is returned in message.reasoning; the answer is in message.content. For coding and math, set presence_penalty to 0.

llama.cpp

Use llama.cpp to run the GGUF checkpoint locally in a terminal or through an OpenAI-compatible API, with optional DFlash2 speculative decoding.

Checkpoint: KRAFTON/Raon-1-mini-preview-gguf

Build from our branch; upstream support is pending. On Windows, see the build guide.

git clone -b raon-1-mini-preview \
    https://github.com/krafton-ai/llama.cpp
cd llama.cpp

# Omit -DGGML_CUDA=ON on Apple silicon.
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Start a terminal chat. The model downloads automatically.

./build/bin/llama-cli \
    -hf KRAFTON/Raon-1-mini-preview-gguf:Q4_K_M \
    --ctx-size 262144 \
    --n-predict 32768 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --repeat-penalty 1.0 \
    --frequency-penalty 0.0 \
    --presence-penalty 1.5 \
    --reasoning on \
    --chat-template-kwargs '{
      "enable_thinking": true,
      "truncate_history_thinking": false
    }'

Or start an OpenAI-compatible server:

./build/bin/llama-server \
    -hf KRAFTON/Raon-1-mini-preview-gguf:Q4_K_M \
    --host 127.0.0.1 \
    --port 8000 \
    --alias Raon-1-mini-preview \
    --ctx-size 262144 \
    --parallel 1 \
    --n-predict 32768 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --repeat-penalty 1.0 \
    --frequency-penalty 0.0 \
    --presence-penalty 1.5 \
    --reasoning on \
    --chat-template-kwargs '{
      "enable_thinking": true,
      "truncate_history_thinking": false
    }'

For coding and math, use --presence-penalty 0.

For DFlash2 speculative decoding, append these options to either command with a trailing \ on the preceding line. The draft model downloads automatically.

    --spec-type draft-dflash \
    --spec-draft-n-max 7 \
    --spec-draft-sampling probabilistic

Send a request from another terminal:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Raon-1-mini-preview",
      "messages": [
        {
          "role": "user",
          "content": "Tell me about KRAFTON AI."
        }
      ],
      "max_tokens": 32768,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 20,
      "min_p": 0.0,
      "repeat_penalty": 1.0,
      "frequency_penalty": 0.0,
      "presence_penalty": 1.5,
      "chat_template_kwargs": {
        "enable_thinking": true,
        "truncate_history_thinking": false
      }
    }'

Reasoning is returned in message.reasoning_content; the answer is in message.content. For coding and math, set presence_penalty to 0.

MLX (Apple silicon)

Use MLX to run Raon-1 mini locally on Apple silicon, either directly in Python or through an OpenAI-compatible API.

Checkpoint: KRAFTON/Raon-1-mini-preview-mlx

Install the dependencies with Python 3.11+:

python3 -m venv .venv
source .venv/bin/activate

python -m pip install \
    "mlx==0.32.4" \
    "mlx-lm==0.32.2" \
    "transformers==5.20.1"

Run in Python:

from mlx_lm import load, generate
from mlx_lm.sample_utils import (
    make_logits_processors,
    make_sampler,
)

model, tokenizer = load(
    "KRAFTON/Raon-1-mini-preview-mlx",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Tell me about KRAFTON AI."}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
    truncate_history_thinking=False,
)

response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=32768,
    sampler=make_sampler(
        temp=1.0,
        top_p=0.95,
        top_k=20,
        min_p=0.0,
    ),
    logits_processors=make_logits_processors(
        repetition_penalty=1.0,
        frequency_penalty=0.0,
        presence_penalty=1.5,
    ),
)
print(response)

Or start an OpenAI-compatible server:

python -m mlx_lm server \
    --model KRAFTON/Raon-1-mini-preview-mlx \
    --trust-remote-code \
    --host 127.0.0.1 \
    --port 8000 \
    --max-tokens 32768 \
    --chat-template-args '{
      "enable_thinking": true,
      "truncate_history_thinking": false
    }' \
    --decode-concurrency 2 \
    --prompt-concurrency 2

Send a request from another terminal:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "default_model",
      "messages": [
        {
          "role": "user",
          "content": "Tell me about KRAFTON AI."
        }
      ],
      "max_tokens": 32768,
      "temperature": 1.0,
      "top_p": 0.95,
      "top_k": 20,
      "min_p": 0.0,
      "repetition_penalty": 1.0,
      "frequency_penalty": 0.0,
      "presence_penalty": 1.5,
      "chat_template_kwargs": {
        "enable_thinking": true,
        "truncate_history_thinking": false
      }
    }'

Use http://localhost:8000/v1 as the base URL and default_model as the model name. For coding and math, set presence_penalty to 0.

Agentic Use

Raon-1 mini preview can be used with agent harnesses such as Hermes Agent, Pi, and OpenCode through an OpenAI-compatible API. The examples below assume an endpoint at http://localhost:8000/v1 with the model ID Raon-1-mini-preview. Adjust these settings to match your server, ensuring that tool calling and reasoning are properly supported.

[!NOTE] With MLX, use KRAFTON/Raon-1-mini-preview-mlx as the model ID instead of Raon-1-mini-preview; mlx_lm server loads the model named in each request.

Hermes Agent

Register Raon-1 mini preview as a custom OpenAI-compatible model in ~/.hermes/config.yaml:

model:
  provider: custom
  default: Raon-1-mini-preview
  base_url: http://localhost:8000/v1
  api_key: dummy

Hermes automatically detects the context window from the server when supported. If detection fails, add context_length: 262144 under model:. Replace dummy with your API key if the endpoint requires authentication.

Pi

Register Raon-1 mini preview as a custom provider in ~/.pi/agent/models.json:

{
  "providers": {
    "raon": {
      "baseUrl": "http://localhost:8000/v1",
      "api": "openai-completions",
      "apiKey": "dummy",
      "models": [
        {
          "id": "Raon-1-mini-preview",
          "name": "Raon-1 mini preview",
          "reasoning": true,
          "contextWindow": 262144,
          "maxTokens": 32768
        }
      ]
    }
  }
}

Select the model using /model inside a session, or launch Pi with --model raon/Raon-1-mini-preview. Replace dummy with your API key if the endpoint requires authentication.

OpenCode

Register Raon-1 mini preview as a custom OpenAI-compatible provider in ~/.config/opencode/opencode.json (or opencode.json in your project root):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "raon": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Raon",
      "options": {
        "baseURL": "http://localhost:8000/v1"
      },
      "models": {
        "Raon-1-mini-preview": {
          "name": "Raon-1 mini preview",
          "limit": {
            "context": 262144,
            "output": 32768
          }
        }
      }
    }
  },
  "model": "raon/Raon-1-mini-preview"
}

The model field sets Raon-1 mini preview as the default model. You can also select it interactively using /models. If the endpoint requires authentication, configure an API key under options or use OpenCode's authentication settings.

For all three agent harnesses, the configured context window and maximum output tokens should not exceed the limits supported by the serving backend.

Access

Raon-1 mini preview is an open-weight model. Access is granted on request.

You can also try it in the Raon-1 mini playground, free for a limited time.

License

Hugging Face model card / README notice License - Apache 2.0

The Raon model, including its model weights, configuration files, and other model components made available for download through this repository, is licensed under the Apache License, Version 2.0. Subject to that license, you may download, use, modify, and redistribute the model and its weights, including for commercial purposes.

Krafton and its licensors retain ownership of their respective intellectual property. The Apache License grants the rights stated in that license and does not transfer ownership. Rights in third-party materials remain subject to their applicable terms.

The Apache License does not extend to Krafton’s proprietary training datasets, training methodologies, undistributed software, hosted API, playground, or other proprietary materials not included in this repository.

Hosted API and Playground Have Separate License Terms

Krafton also makes Raon available through its hosted API and playground service (“RAON Service”). The Raon Service is not licensed under Apache 2.0. Access to and use of the Raon Service is governed by separate Hosted Access Terms and Conditions that are provided before access to the service is granted. The Hosted Access Terms and Conditions govern, among other things, permitted service use, suspension, termination, and Krafton’s right to charge fees in the future.