evalexplorer-classify-gguf

GGUF exports of the EvalExplorer document classifier adapters, for llama.cpp. Each folder holds Q8_0, Q5_K_M and Q4_K_M files of the base model with the LoRA merged in, the LoRA itself as GGUF (*-lora-f16.gguf, to apply at load time with --lora on an unmerged base) and the importance matrix used for the K-quants. Every file carries its base model's licence.

The models classify an international development evaluation report from its first pages into evaluation_approach, evaluation_type, temporality, themes and countries. They were trained on the EvalExplorer ingestion pipeline's LLM labels as a quick exploration of what small models can do; the intended next version is trained on the GLM-5.3-Flash relabelling.

Scores

Test split of baobabtech/evalexplorer-data (config classify_codes, 134 documents). Score is the mean field score against the pipeline labels the models learned; vs GLM scores the same answers against the GLM-5.3-Flash relabelling, which the models never saw. Both label sets are unreviewed LLM output. "With schema" constrains the response to a JSON schema of the allowed codes. Speed: llama-server on one A100, 8 parallel slots. With 134 documents, differences below about 0.03 are within sampling noise.

qwen3.5-2b-grpo-countries

Base unsloth/Qwen3.5-2B, adapter baobabtech/evalexplorer-classify-adapters/qwen3.5-2b-grpo-countries.

qwen3.5-4b-sft

Base unsloth/Qwen3.5-4B, adapter baobabtech/evalexplorer-classify-qwen3.5-4b-sft.

gemma-4-e2b-grpo-lr5e6

Base unsloth/gemma-4-E2B-it, adapter baobabtech/evalexplorer-classify-adapters/gemma-4-e2b-grpo-lr5e6.

gemma-4-26b-a4b-sft

Base unsloth/gemma-4-26B-A4B-it, adapter baobabtech/evalexplorer-classify-gemma-4-26b-a4b-sft.

Every row links to a full report in baobabtech/evalexplorer-classify-experiments (runs named <adapter>--gguf-<quant>[-lora][-schema]--test).

Findings

• Q8_0 matches the PyTorch scores within 0.006 for the dense models (Qwen3.5 2B and 4B, Gemma 4 E2B).
• Q4_K_M costs about 0.02 for the 2B-class models and less than 0.01 for Qwen3.5-4B.
• The JSON schema changes the scores by less than 0.005 for the dense models: they already return valid JSON.
• Gemma 4 26B-A4B (MoE) scores about 0.05 below its PyTorch run, mostly on evaluation approach. The GGUF is faithful to the adapter as plain transformers applies it: there, the same adapter scores 0.796 unmerged and 0.794 merged (jobs/merge_check.py), against 0.790 for Q8_0. The 0.844 PyTorch score was measured under Unsloth, whose Gemma 4 MoE path applies the expert LoRA differently. Its LoRA sits on the expert tensors, which llama.cpp's LoRA converter cannot map, so the adapter was merged with PEFT first and there is no LoRA reference row.

Usage

llama-server -m gemma-4-e2b-grpo-lr5e6/gemma-4-e2b-grpo-lr5e6-Q4_K_M.gguf -c 8192

Render the prompt from the dataset's prompt column with the base model's chat template and thinking off (enable_thinking=False), send it to /completion with temperature: 0, and parse the JSON answer. To constrain the output, pass json_schema with the allowed codes per field; jobs/gguf.py in code/ builds it (json_schema) and has the full export and scoring pipeline.

Built with llama.cpp b11361: convert_hf_to_gguf.py and convert_lora_to_gguf.py, llama-export-lora, llama-imatrix on 200 training documents, llama-quantize.