NeoHorse-1-4B — CMF

NeoHorse-1-4B (Apache-2.0) packaged as a single CMF file. It runs with cortiq, a Rust inference engine with no Python and no ML framework: NVIDIA, AMD and Intel GPUs through Vulkan/DX12, Apple silicon through Metal, and the CPU.

The model is a 4.21B text-only fine-tune of Qwen3.5-4B: 24 linear-attention layers interleaved with 8 full-attention layers (one every fourth layer), 256k declared context, agentic/tool-use post-training. It is text-only — the Qwen3.5 vision tower is not part of this checkpoint.

English · Русский · 中文

Quick start

cargo install cortiq-cli
hf download infosave/NeoHorse-1-4B-cmf neohorse-1-4b-q4tp.cmf --local-dir .
cortiq run neohorse-1-4b-q4tp.cmf --prompt "Explain quicksort in three sentences." --no-think

Prebuilt binaries for Linux, macOS and Windows are on the releases page. The engine picks the GPU and its settings itself. --no-think asks for a direct answer; without it the model reasons first, so raise --max-tokens (default 256).

Files

file	quantization	size	wikitext-2 ppl	recommended for
`neohorse-1-4b-q4tp.cmf`	4-bit, ladder scales	2.22 GB	13.42	default: laptops, 6–8 GB VRAM, CPU inference
`neohorse-1-4b-q8_2f.cmf`	8-bit	4.24 GB	12.66	highest precision, 8 GB VRAM or more

Both profiles are quantized directly from the bf16 checkpoint in one streamed pass. Perplexity protocol: twelve 512-token windows of wikitext-2 test, exact attention, scored on the RTX 5080 with the same binary as the speed numbers.

Layout: q8_2f profile = 201 q8_2f matrices + 225 f16 embeddings/norms/head entries; q4tp = 4-bit nibbles with a per-row geometric scale ladder.

Performance

Decode speed of one stream, tok/s (cortiq bench --core --ignore-eos):

hardware	backend	file	decode	prompt processing	time to first token
RTX 5080 (16 GB)	Vulkan	`q4tp`	165.0	685	0.06 s
RTX 5080 (16 GB)	Vulkan	`q8_2f`	109.4	261	0.16 s
16-core x86 host	CPU	`q4tp`	17.1	69.2	0.59 s
6-core x86 host	CPU	`q4tp`	5.9	14.2	2.94 s
6-core x86 host	CPU	`q8_2f`	4.5	22.9	1.70 s

Long context (cortiq bench --ctx N, RTX 5080, Vulkan):

prompt	file	prompt processing, tok/s	time to first token	decode at this depth, tok/s
1000 tokens	`q4tp`	1330	0.75 s	156.0
1000 tokens	`q8_2f`	299	3.34 s	102.2
16000 tokens	`q4tp`	801	20.0 s	107.2
16000 tokens	`q8_2f`	260	61.5 s	78.5

Notes measured on this build:

Do not enable --o1 on this model. The Nyström O(1) path replaces exact KV attention on full-attention layers; with 24 of 32 layers already linear attention it only adds work — decode fell from 107 to 33 tok/s at 16k context on the RTX 5080.
No speculative decoding: the checkpoint's header declares an MTP head but the released weights carry none, so the engine loads without it.
On CPU the optimum is 5–6 threads on a 6-core host; 12 threads (SMT) is ~20% slower. q4tp decodes 30% faster than q8_2f but prefills 1.6× slower — activation quantization of 4-bit weights costs more per prefill token.

Usage

cortiq run neohorse-1-4b-q4tp.cmf                           # interactive chat
cortiq run neohorse-1-4b-q4tp.cmf --prompt "..."            # one answer
cortiq run neohorse-1-4b-q4tp.cmf --prompt "..." --no-think # answer without the thinking block
cortiq run neohorse-1-4b-q4tp.cmf --prompt "..." --greedy   # deterministic output

Sampling

Upstream's reported protocol (thinking mode):

mode	temperature	top_p	top_k	min_p	presence_penalty	repetition_penalty
thinking	1.0	0.95	20	0.0	1.5	1.0
direct (`--no-think`)	0.7	0.80	20	0.0	1.5	1.0

Flags: --temperature, --top-p, --top-k, --min-p, --presence-penalty, --rep-penalty, --seed, --max-tokens. Without flags the CLI uses temperature 0.7, top-p 0.9, top-k 40, min-p 0.05, repetition penalty 1.1 and at most 256 new tokens.

OpenAI-compatible server

cortiq serve neohorse-1-4b-q4tp.cmf --port 8080

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "neohorse-1-4b-q4tp", "messages": [{"role": "user", "content": "Hi!"}]}'

Endpoints: /v1/chat/completions, /v1/completions, /v1/models.

Quality battery

Greedy, --no-think, CPU 6 cores and RTX 5080 Vulkan; a cell quotes the GPU answer only where it differs from the CPU one:

check	`q8_2f` CPU	`q8_2f` Vulkan	`q4tp` CPU	`q4tp` Vulkan
`17 * 23`	391 ✓	391 ✓	391 ✓	391 ✓
year WWII started (Russian)	1939 ✓	1939 ✓	1939 ✓	1939 ✓
Python one-line factorial, code only	`fact = lambda n: 1 if n <= 1 else n * fact(n - 1)` ✓	`f=lambda n:1 if n<2 else n*f(n-1)` ✓	`return math.factorial(n)` ✗	`f=lambda n:int(sum(range(1,n+1)))*int(flam(n-1))if n>0else 1` ✗
syllogism with undistributed middle (roses/flowers)	«да» ✗	«Да» ✗	«да» ✗	«да» ✗
exactly two sentences about Moscow	2 sentences ✓	2 sentences ✓	2 sentences ✓	2 sentences ✓
three hash-table uses in one sentence	✓	✓	✓	✓

Known limitations, kept as measured:

Both profiles answer "yes" to the roses/flowers syllogism.
The 4-bit profile degrades on one-line code formatting while keeping the arithmetic correct; the 8-bit profile produces a valid lambda on CPU and GPU.
Greedy output is identical between the CPU and Vulkan backends for the 4-bit profile (verified on a 40-token answer, byte-for-byte).

Hardware

q4tp 2.22 GB: from 3 GB of VRAM/RAM; the default for laptops and CPU-only hosts.
q8_2f 4.24 GB: from 5 GB; the precision choice where memory allows.
On Vulkan the whole 2.22 GB (or 4.24 GB) weight set is uploaded once and stays resident; the first token after load pays a one-time upload (0.89 s for q4tp, 1.03 s for q8_2f on the RTX 5080, pipelines cached afterwards).
Declared context is 262,144 tokens; the linear-attention core keeps state O(1) per layer and only the 8 full layers grow a KV cache.

Verify

sha256sum -c neohorse-1-4b-q4tp.cmf.sha256
cortiq verify neohorse-1-4b-q4tp.cmf      # envelope, sections, per-tensor hashes
cortiq info   neohorse-1-4b-q4tp.cmf      # arch, layers, params

Expected: qwen3_5_text | 426 tensors | 4.21B params, verify → OK on both files.

Документация на русском

Что это. NeoHorse-1-4B (Apache-2.0) — текстовый fine-tune Qwen3.5-4B: 24 слоя linear-attention + 8 слоёв полного внимания (каждый четвёртый), 262k контекста, agentic/tool-use обучение. Упакован в один файл CMF и запускается движком cortiq на Rust — без Python и ML-фреймворков: Vulkan/DX12 (NVIDIA, AMD, Intel), Metal (Apple), CPU.

Файлы. q4tp — 2.22 ГБ, ppl 13.42, по умолчанию; q8_2f — 4.24 ГБ, ppl 12.66, максимальная точность. Протокол ppl: двенадцать окон по 512 токенов wikitext-2, точное внимание, RTX 5080.

Скорость (один поток). RTX 5080 Vulkan: q4tp — 165 tok/s decode, 685 tok/s prefill, первый токен 0.06 с; q8_2f — 109 / 261 / 0.16 с. Контекст 16k: q4tp — 107 tok/s при prefill 801 tok/s, TTFT 20 с; q8_2f — 78.5 tok/s, TTFT 61 с. CPU: 6 ядер — 5.9 tok/s (q4tp) и 4.5 (q8_2f); 16 ядер — 17.1 tok/s.

Важно. Не включайте --o1 на этой модели: 24 из 32 слоёв уже linear-attention, надстройка Nyström только замедляет (107 → 33 tok/s на 16k). Спекулятивного декодирования нет — MTP-голова в заголовке объявлена, но весов в релизе нет.

Проверка. sha256sum -c neohorse-1-4b-q4tp.cmf.sha256, затем cortiq verify neohorse-1-4b-q4tp.cmf → OK.

Лицензия. Apache-2.0, как у апстрима. Апстрим-модель — Qwen/Qwen3.5-4B (Copyright 2026 Alibaba Cloud); TokenRhythm дообучил и переупаковал её для текстового инференса. Настоящая карточка описывает только переупаковку в CMF.

中文文档

简介。 NeoHorse-1-4B (Apache-2.0) 是 Qwen3.5-4B 的纯文本微调版本：24 层线性注意力 + 8 层全注意力（每四层一个），声明上下文 262k，面向 agentic / tool-use。已打包为单个 CMF 文件，由 Rust 推理引擎 cortiq 运行，无需 Python 与 ML 框架。

文件。 q4tp — 2.22 GB，ppl 13.42，默认选择；q8_2f — 4.24 GB，ppl 12.66，最高精度。困惑度协议：wikitext-2 测试集，12 × 512 token 窗口，精确注意力， RTX 5080 实测。

速度（单流）。 RTX 5080 + Vulkan：q4tp 解码 165 tok/s、预填充 685 tok/s、首字 0.06 s；q8_2f 109 / 261 / 0.16 s。16k 上下文：q4tp 解码 107 tok/s、预填充 801 tok/s；q8_2f 解码 78.5 tok/s、首字 61 s。CPU：6 核 5.9 tok/s （q4tp）与 4.5 tok/s（q8_2f）；16 核 17.1 tok/s。

注意。 不要对该模型启用 --o1：32 层中已有 24 层是线性注意力，Nyström 路径只会更慢（16k 上下文从 107 降到 33 tok/s）。无投机解码：头部声明了 MTP，但发布权重中不含该部分。

校验。 sha256sum -c neohorse-1-4b-q4tp.cmf.sha256，再运行 cortiq verify neohorse-1-4b-q4tp.cmf，应输出 OK。