NeoHorse-1-4B-GGUF

Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness.

Technical Report

NeoHorse-1-4B is a 4B causal language model and an initial prototype on the path toward recursive self-improvement (RSI). It is post-trained from Qwen3.5-4B for text-based agent harnesses, tool use, coding, and instruction following.

This repository provides GGUF versions of NeoHorse-1-4B for local use with llama.cpp, Ollama, and LM Studio. It includes 16-bit (BF16) weights and smaller 8-bit, 5-bit, and 4-bit quantized versions. Quantized versions take up less disk space and use less memory, making the model easier to run on your own hardware. These files contain text-only model weights, fine-tuned by TokenRhythm from Qwen3.5-4B.

Highlights

• Path toward RSI: the routing harness assigns tasks to a heterogeneous model pool, records tool interactions and outcomes, estimates capability demand, and uses capability-level feedback to shape the next training mixture. Updated models can return to the harness, closing a prototype evaluation–selection–update loop; extending this loop across successive iterations is the next step toward RSI.
• Agentic post-training framework: the associated research explores routing-guided curriculum SFT and routing-guided on-policy distillation to turn execution trajectories into training signal while preserving execution and harness context around each response.
• Data quality: exact and near-duplicate removal, evaluation decontamination, structural validation, six-dimensional semantic evaluation, and subscene-level Scene/Goal/Outcome labeling.
• Broad gains: 64.87 macro average across ten benchmarks versus 58.94 for Qwen3.5-4B (+5.93).

Model Details

Evaluation

The results below are from the original checkpoint, not a separate GGUF benchmark run.

The 4B track compares NeoHorse-1-4B with five representative open-weight models. Results are grouped by capability in the table below. Higher is better; Δ is NeoHorse-1-4B minus Qwen3.5-4B. Bold marks the best available result; underlining marks the second-best.

* Nanbeige-4.2-3B LiveCodeBench v6 result is reported in the corresponding model's official blog post or technical report.

Reported protocol: SGLang v0.5.17 · temperature=1.0 · top_p=0.95 · top_k=20 · min_p=0.0 · presence_penalty=1.5 · repetition_penalty=1.0 · thinking mode enabled with enable_thinking=true and force_nonempty_content=true. QwenClawBench, WorkBuddy Bench, and tau2-Bench use three runs; PinchBench and VitaBench use one run; the remaining benchmarks follow their official protocols. VitaBench uses the DeepSeek-V4-Flash simulator and judge.

Deployment

FP16 option: NeoHorse-1-4B-F16.gguf (8.42 GB) is also available for runtimes or hardware that prefer FP16 over BF16. Use :F16 instead of :BF16 in the llama.cpp or Ollama commands below. This is a format-converted 16-bit option, not a quality upgrade.

Quantized files

4-bit is the smallest option, 5-bit sits in the middle, and 8-bit uses less compression. Choose 16-bit (BF16) to keep the original weight precision. Lower precision saves memory but can affect response quality. The sizes above are file sizes; running the model also needs memory for the conversation and other runtime data.

These standard llama.cpp quantizations were generated directly from the BF16 GGUF, without an importance matrix. All three passed eight short llama.cpp compatibility checks, including thinking, single and parallel tool calls, and tool-result continuation, at 4K context. These checks are not a quantized-quality benchmark; the benchmark scores in this card refer to the original checkpoint. Quantized variants have not yet been separately tested in Ollama or LM Studio. Use the desired filename in place of the BF16 filename in the examples below.

Use a recent runtime with Qwen3.5 support. The commands below download the model from Hugging Face and cache it locally; no manual file download is required. These are text-only GGUF files with an embedded chat template and no MTP draft head.

llama.cpp

Download and start a local OpenAI-compatible server:

llama-server -hf TokenRhythm/NeoHorse-1-4B-GGUF:BF16 -c 4096 -ngl 99 --jinja --port 8080 --alias neohorse-1-4b

Ollama

Download and run directly, without creating a Modelfile:

ollama run hf.co/TokenRhythm/NeoHorse-1-4B-GGUF:BF16

LM Studio

Paste this repository URL into LM Studio's model search and choose a GGUF file, or download through the CLI:

lms get https://huggingface.co/TokenRhythm/NeoHorse-1-4B-GGUF

Select the desired file when prompted, then load it in LM Studio with MTP speculative decoding disabled. Start with a 4K context and increase it according to available memory. In LM Studio's native API, use reasoning: "off" to disable thinking.

For llama.cpp and Ollama, replace :BF16 with the tag of another variant listed in Files and versions, such as :Q8_0, :Q5_K_M, or :Q4_K_M. Only variants already present in the repository can be downloaded.

See the llama.cpp guide, Ollama–Hugging Face integration, and LM Studio download guide.

Manual download and local-file setup (optional)

Download NeoHorse-1-4B-BF16.gguf (approximately 8.42 GB). Use a recent runtime with Qwen3.5 support. The file includes the chat template and does not include an MTP draft head.

llama.cpp

llama-server -m NeoHorse-1-4B-BF16.gguf -c 4096 -ngl 99 --jinja --port 8080 --alias neohorse-1-4b

Ollama

Create a Modelfile containing:

FROM ./NeoHorse-1-4B-BF16.gguf
PARAMETER num_ctx 4096

ollama create neohorse-1-4b-bf16 -f Modelfile
ollama run neohorse-1-4b-bf16

LM Studio

Import the GGUF file and load it with MTP speculative decoding disabled. With the CLI:

lms import ./NeoHorse-1-4B-BF16.gguf --copy
lms load neohorse-1-4b --context-length 4096 --no-speculative-draft-mtp

Basic chat and tool-call smoke tests passed in llama.cpp, Ollama 0.33.3, and LM Studio's llama.cpp 2.33.0 engine. Context capacity depends on available memory; the examples start at 4K. In LM Studio's native API, use reasoning: "off" to disable thinking.

Using NeoHorse-1-4B-GGUF via the Chat Completions API

After starting llama-server as shown in Deployment, connect an OpenAI-compatible client to http://127.0.0.1:8080/v1. The examples below use the server alias neohorse-1-4b; it must match the --alias value in the launch command.

Basic Usage

Install the Python client:

pip install openai

Send a message to your local server:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="EMPTY",  # Replace if server authentication is enabled.
)

generation_config = {
    "temperature": 1.0,
    "top_p": 0.95,
    "presence_penalty": 1.5,
    "max_tokens": 2048,
    "extra_body": {
        "top_k": 20,
        "min_p": 0.0,
        "repeat_penalty": 1.0,
        "chat_template_kwargs": {"enable_thinking": True},
    },
}

response = client.chat.completions.create(
    model="neohorse-1-4b",
    messages=[
        {
            "role": "user",
            "content": "Write a Python function that returns the first n Fibonacci numbers.",
        }
    ],
    **generation_config,
)

message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

These short examples allow up to 2,048 output tokens within the 4K server context. For longer prompts or responses, increase -c and adjust max_tokens while leaving room for the input and chat template.

Tool Calling

Keep --jinja enabled in the llama-server launch command so it can use the embedded tool-aware chat template and parse generated function calls into tool_calls.

Using the same client and generation_config from Basic Usage:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather for a specified city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "Name of the city.",
                    }
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="neohorse-1-4b",
    messages=[
        {
            "role": "user",
            "content": "What is the weather in Beijing right now?",
        }
    ],
    tools=tools,
    tool_choice="auto",
    **generation_config,
)

message = response.choices[0].message

if message.tool_calls:
    for call in message.tool_calls:
        print(call.function.name, call.function.arguments)
else:
    print(message.content)

This example prints the requested function and its arguments. Your application executes the function and returns its result in a tool message, using the corresponding tool_call_id, to continue the conversation.

License

NeoHorse-1-4B is released under the Apache License 2.0.

The upstream model is Qwen/Qwen3.5-4B. Its original copyright notice, Copyright 2026 Alibaba Cloud, is retained in the license file. TokenRhythm has modified the model through fine-tuning and repackaging for text-only inference. This repository converts the resulting checkpoint to BF16 GGUF and provides quantized variants using llama.cpp.

Citation

@misc{neohorse2026,
  title        = {NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness},
  author       = {NeoHorse Team},
  year         = {2026},
  howpublished = {arXiv preprint},
  eprint       = {2609.08183},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2609.08183}
}

For questions or issue reports, use the NeoHorse project repository.