NeoHorse-Jev-4B-GGUF

Prefill-only decisions for agent workflows · GGUF edition

Original Model · GGUF Files · Deployment

Introduction

NeoHorse-Jev-4B-GGUF is the GGUF release of NeoHorse-Jev-4B, a 4B structured decision model from TokenRhythm, built on NeoHorse-1-4B. Given a state and application-defined questions, it uses prefill-only inference to predict decisions and their probabilities for routing requests, selecting tools, checking conditions, and rating outcomes.

This repository provides BF16 weights and Q8_0, Q4_K_M, Q3_K_M, and Q2_K quantized downloads. Each GGUF contains the language backbone, visual weights, decision head, and tokenizer configuration. Download one GGUF plus the shared runtime/ directory; no separate mmproj or original model weights are needed. The supplied runtime supports text requests and requests combining one image with text.

For the full model introduction, comparisons, and usage guide, see the original model. This page covers GGUF quantization results, file selection, and deployment.

Decision Demos

Six decision demos: Tetris, Snake, robot manipulation, Mahjong, four-player bomb arena, and autonomous driving, from left to right and top to bottom. Each panel preserves the original replay and decision displays and loops independently.

These decision demos feature the original NeoHorse-Jev-4B.

Evaluation

GGUF Quantization Results

Start with Q4_K_M: its complete file is 3.39 GB, about 63% smaller than BF16, with a three-benchmark average of 83.77%.

Scores are accuracy (%). AVG is the unweighted mean of Nimble, VitaminC, and MASSIVE, evaluated on 282, 599, and 350 examples respectively. Bold marks the best score in each column; underline marks the second-best. File sizes use decimal GB.

Original model comparisons

The following results are from the original model card, updated September 24, 2026. NeoHorse-Jev-4B scores 77.70 on the six-group text aggregate, the highest among the four open-weight decision models with complete results in this comparison. Its mean accuracy across Nimble, VitaminC, and MASSIVE is 83.26%, 11.50 percentage points above NeoHorse-1-4B.

The original model uses vLLM for JevBench, Kev, and OpenJev; the other three benchmarks retain the fixed-subset evaluation results.

Bold and underline mark the best and second-best scores among the listed open-weight models; -- means no result is available. AVG weights the six groups equally and ranks only models with complete results; image and game results are excluded. See the original evaluation section for protocols, detailed results, image understanding, and interactive tasks.

Download

Download Q4_K_M and its supporting files from Hugging Face:

pip install huggingface_hub
hf download TokenRhythm/NeoHorse-Jev-4B-GGUF \
  --include 'NeoHorse-Jev-4B-Q4_K_M.gguf' 'runtime/*' 'LICENSE*' 'README.md' 'SHA256SUMS' \
  --local-dir neohorse-jev-gguf

Replace Q4_K_M with another linked precision from the table to use that version. Each GGUF file is a complete alternative, not a shard; choose one.

Deployment

Use the NeoHorse GGUF runtime included in this repository. It reads all model weights from the same GGUF and provides Jev's Python decision interface. The included llama.cpp-based decision adapter supports local Python and command-line inference. For vLLM, SGLang, or native HTTP deployment, use the original model and its deployment guide.

Build and Run with llama.cpp

Tested environment: Linux x86-64 with an NVIDIA SM90 GPU, Python 3.12, CUDA toolkit 12.4, GCC 11, PyTorch 2.8.0, and Transformers 5.17.0. Install git, CMake, a C++ compiler, and the CUDA toolkit, then build from source for the target GPU.

cd neohorse-jev-gguf
python -m venv .venv
source .venv/bin/activate
pip install -r runtime/requirements.txt

# Set this to your GPU's compute capability; 90 is the tested configuration.
python runtime/build.py --cuda-arch 90

# Run the bundled text decision example.
CUDA_VISIBLE_DEVICES=0 python runtime/example.py \
  --model NeoHorse-Jev-4B-Q4_K_M.gguf

The build downloads a pinned llama.cpp revision. Inference uses the local GGUF and requires no base-model download.

Quick Decisions with Python

One user message, three decisions: a yes/no probability, a selected option, and an ordered rating. After building the runtime, run this example from neohorse-jev-gguf/. Text requests can contain multiple questions; the current GGUF runtime evaluates them sequentially.

import sys
sys.path.insert(0, "runtime")
from runtime import NeoHorseGGUF

model = NeoHorseGGUF("NeoHorse-Jev-4B-Q4_K_M.gguf")
try:
    result = model.predict({
        "state": "I was charged twice for the same order. Please refund the extra charge today.",
        "questions": {
            "refund": {
                "type": "noul",
                "instructions": "Is the user requesting a refund?"
            },
            "team": {
                "type": "choice",
                "instructions": "Which team should handle this message?",
                "criteria": {
                    "billing": "Bills, charges, or refunds",
                    "technical": "Product failures or technical issues",
                    "other": "Other matters"
                }
            },
            "urgency": {
                "type": "score",
                "instructions": "How soon does the user want this resolved?",
                "criteria": ["Can wait", "This week", "Today"]
            }
        }
    })
    answers = result["answers"]
    print("Noul / refund probability:", answers["refund"]["noul"])
    print("Choice / team:", answers["team"]["choice"])
    print("Candidate probabilities:", answers["team"]["probabilities"])
    print("Score / urgency:", answers["urgency"]["score"])
finally:
    model.close()

Noul returns P(true). Choice returns the selected option and its probability distribution. Score returns the expected level, weighted by the level probabilities. The three criteria above have indices 0, 1, and 2, and the score can be fractional. Applications can read these fields directly to set routing and priorities.

Image and Text Decisions

The bundled example selects the dominant color of a local image from a fixed candidate set:

CUDA_VISIBLE_DEVICES=0 python runtime/example.py \
  --model NeoHorse-Jev-4B-Q4_K_M.gguf --image example.png

For a custom request, pass a local PIL.Image as the second argument to model.predict(request, image). Image mode accepts one image and one question. See runtime/example.py for the complete request.

Runtime limits and implementation details

• Text: state up to 384 tokens; packed branch per question up to 1,024 tokens; packed input up to 2,048 tokens.
• Image: one image up to 4 megapixels; 64–1,024 visual tokens; state up to 2,048 tokens; packed text up to 8,192 tokens; expanded context up to 12,288 tokens. Oversize inputs are rejected.
• Serving: one call per instance at a time; close the model to release GPU memory. Switching between text and image modes reloads the native engine.
• Precision: only the language backbone is quantized. Visual weights remain BF16 with selected FP32 tensors; the decision head remains FP32. Q4_K_M, Q3_K_M, and Q2_K share a training-input importance matrix; BF16 and Q8_0 do not use it. No visual fine-tuning was performed.
• Build: llama.cpp is pinned to 94256114c229674ef96e76eb2dea596e65b43818. The supplied loader change skips visual and decision-head namespaces in the language loader; the visual loader and Python readout access those tensors from the same GGUF. No second weight file is extracted; only small tokenizer configuration files are temporarily materialized.

License and Acknowledgments

Model weights are released under Apache-2.0. NeoHorse-Jev builds on NeoHorse-1-4B; this repository converts and packages the Jev model for GGUF inference.

We thank Jared Palmer for open-sourcing Kev, from which parts of the decision runtime are adapted, and llama.cpp for the GGUF inference foundation. Their accompanying license and attribution notices are preserved. See SHA256SUMS for file integrity checks.

For questions and bug reports, use NeoHorse Issues.