SHENNONG 神农 — 1.5 B domain fine-tune

This repository contains three different things. Read the table before downloading.

The main artifact is a LoRA fine-tune of Qwen/Qwen2.5-1.5B on 88,082 domain examples about classical Asian medicine, merged back into the base weights and also provided as a GGUF for CPU inference. The same repo also holds two sparse-mixture-of-experts experiments that are research artifacts, not the recommended model — one of them lost to the dense model, and one of them requires a specific llama.cpp build.

What is in this repository

file	size	what it is	use it?
`model.safetensors`	3,087 MB	The model. Full merged fp16 weights, 1,543,714,304 parameters. LoRA r=64 merged into Qwen2.5-1.5B.	✅ transformers
`shennong-1.5b-sft-Q4_K_M.gguf`	986 MB (940 MiB)	The recommended GGUF. Q4_K_M quantisation of the above, runs on CPU with llama.cpp, no GPU needed.	✅ llama.cpp
`adapters/t2_natprod.gguf`	39 MB	A LoRA adapter (Track-02 natural-products domain) for the 0.5 B model, used by an expert-routing demo. Optional.	optional
`shennong-moe-Q4_K_M-loadable.gguf`	1,368 MB	4-expert top-2 sparse MoE, 1,435,499,392 params, repaired to load in llama.cpp 0.3.x. Scored below the dense model (31.7 % vs 43.3 %).	research only
`shennong-moe-Q4_K_M.gguf`	1,047 MB	The same MoE before the repair. Its shared-expert tensors have a 0-width placeholder shape, so llama.cpp 0.3.x rejects it. Kept for provenance.	❌ will not load

Tokenizer and chat-template files are included.

Training

Base: Qwen/Qwen2.5-1.5B (Apache-2.0)
Method: LoRA, r=64, alpha=128, dropout 0.05, targets q_proj k_proj v_proj o_proj gate_proj up_proj down_proj
Data: 88,082 domain examples (SFT), 5,623 held out — question/answer pairs over classical Chinese materia medica, plant-name resolution, compound classes, gene targets, formulas, incompatibilities and safety
Steps: 1,800 (effective batch 16 → ≈0.33 epoch)
Final losses: train 1.187 · eval 1.031
Merge: adapters merged into the base, then quantised to Q4_K_M

Evaluation

200 held-out questions, same decode for both arms (temperature 0.2, top_p 0.9, seed 1337, max_new_tokens 40), contains-match of the gold fact, no retrieval. Full protocol and the benchmark's own defects are documented in docs/BENCHMARK.md.

arm	score
this model	49.0 % (98/200)
`Qwen/Qwen2.5-1.5B-Instruct` (un-finetuned)	37.0 % (74/200)

Per task family (fine-tuned vs base): compound 20/20 vs 20/20 · target 20/20 vs 20/20 · pharmacology 20/20 vs 20/20 · viability 6/6 vs 0/6 · property 15/20 vs 0/20 · formula 7/9 vs 5/9 · name_mapping 8/20 vs 7/20.

What this model is bad at — measured, not guessed

use questions: 0/63. Asked "what is Echinochloa colona used for in Indian medicine?" it answers with a Chinese classical use. It confuses traditions. Do not use it to recall therapeutic uses; use the extraction pipeline in the repository instead.
Name resolution is weak (8/20). Archaic plant names outside the lexicon are not resolved.
Greek and Latin are not represented. The corpus holds ~19 records' worth; the model cannot decode them.
The sibling sparse MoE scored 31.7 % vs 43.3 % for the dense model — it is published as a negative result.

How to use

CPU, via llama.cpp (recommended):

llama-cli -m shennong-1.5b-sft-Q4_K_M.gguf -p "<|user|>What gene target does Salvia miltiorrhiza affect?<|assistant|>" -n 40

transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("cptlight/shennong-1.5b-sft")
m = AutoModelForCausalLM.from_pretrained("cptlight/shennong-1.5b-sft", torch_dtype="float16")
out = m.generate(**tok("<|user|>What is 甘草 used for?<|assistant|>", return_tensors="pt"), max_new_tokens=40)
print(tok.decode(out[0], skip_special_tokens=True))

Licence and provenance

Apache-2.0 (inherited from Qwen2.5-1.5B, which is Apache-2.0; our fine-tuning and code are Apache-2.0). Training data derives only from permissive sources — CC0, public domain and CC BY. Sources whose licences are non-commercial (ctext.org, GRETIL) were excluded, which is recorded source-by-source in docs/CORPUS_PROVENANCE.md.

Intended use and limits

This is a research model for the history and pharmacology of traditional medicine. It answers questions about recorded uses, historical plant names, compound classes and gene targets, and it summarises retrieved passages.

It is not medical advice. It does not provide dosing, does not recommend self-treatment, and does not give synthesis routes for toxins or pathogens. Its outputs are pre-modern claims about a historical record, not clinical guidance, and nothing here has been validated in vitro or in vivo.

cptlight/shennong-1.5b-sft