Theo Qwen Base

theo-qwen-base is a 1.5B-parameter base (pretrained, not instruction-tuned) causal language model for biblical and theological English. It started from Qwen/Qwen3-1.7B. Qwen's vocabulary was swapped for a 32,788-token Bible-tuned one, value-embedding residuals were added on 11 layers, and the model was then continued-pretrained on about 71M tokens of grounded Bible, theology and general English text.

This is the checkpoint the instruction-tuned theo-qwen-instruct was fine-tuned from. Use this one for plain text continuation, log-likelihood scoring, or as a starting point for your own fine-tuning.

What This Model Is For

Recommended use cases:

• Continuing biblical, devotional and theological text
• Scoring candidate sentences or completions by log-likelihood (cloze, ranking, reranking)
• A base for supervised or preference fine-tuning in the Bible/theology domain

It is a base model: it continues text and does not follow instructions or chat turns. It has no chat template.

How It Was Built

Qwen/Qwen3-1.7B @70d244cc                                  1.72B params, 151,936-token vocab
 └─ Bible devotional LoRA merged in (71,644 pairs, 1 epoch)
     └─ transplant: vocab shrink 151,936 → 32,788, + value-embedding residuals    1.50B params
         └─ grounded continued pretraining, ~71M tokens, 4,334 steps    ← this release

1. LoRA merge. A rank-16 LoRA on every attention and MLP projection was trained for one epoch on 71,644 Bible devotional instruction pairs and merged into Qwen's weights. Merging changes values, not shapes.

2. Transplant. The merged model's 28 layers, attention heads, Q/K norms and final norm were copied unchanged, giving 1,409,410,048 core parameters, the same as Qwen. The embedding was rebuilt for a new 32,768-token byte-level BPE trained on Bible-domain and general English text, plus [unused1]–[unused20] reserved tokens. 28,355 rows are exact copies of a Qwen token's row and 4,260 are the mean of the 2–5 Qwen rows that spell the token. Value-embedding residuals were added to layers 0, 2, …, 20, initialized inert (alpha 0). Fresh from the transplant, the model scored 41.1 perplexity on C4 validation.

Cedar is the author's internal name for the training code, tokenizer and checkpoints behind this model family. It appears in run names, in the tokenizer's name, and below as the "previous Cedar base": an earlier internal base, never released, that was continued-pretrained on an older, ungrounded synthetic corpus.

3. Grounded continued pretraining (CPT) — this release. A planned mix of 71,000,000 tokens, one pass, sequence length 1024. The final checkpoint had consumed 70,749,535 of them:

The mix included general English material, Bible and theology-domain text, generated and curated passage-grounded explanations, attributed reading notes, grounded Q&A, and Scripture-related text. Individual sources and per-category token counts are not listed here.

Each document was framed as <bos> + text + <eos>. The design goal was a base model that says only what its source says: it should keep speakers, hedges and attributions, and stop when it is done. The earlier synthetic corpus had taught the opposite. Every training document was checked for leakage against the evaluation sets below. For the theology ruler the check was a 32-token overlap rule, so short verse quotations can still appear in training.

Hyperparameters: Adafactor at lr 5e-5 with 500k warmup tokens and a cosine schedule to 0.1×, weight decay 0.1, clip 1.0, 16,384 tokens per step, bf16, one RTX 4090 for about 4 hours. Final C4 validation loss was 2.899 (perplexity 18.15).

Evaluation

Theology ruler (545 cases)

The same 546-case theological cloze set as theo-bert-base. One case has no [MASK] and is excluded. A causal LM has no [MASK], so each candidate word is substituted into the sentence, every resulting sentence is scored by log-likelihood, and the case's pass condition is applied to that ranking. Details are in EVAL.md.

These numbers are not comparable to theo-bert-base's. That model is scored on its top-5 over the full vocabulary; this one ranks only each case's 5–10 candidates.

All three columns were scored with the same script on the same GPU. The raw-transplant column is from the internal build; its per-case results are not shipped here.

By difficulty:

By category:

Contrastive theology, where a correct word competes with a deliberately plausible wrong one, is where the donor is weakest and where CPT helped most. The median margin of correct over foil grew from 8.0 nats (Qwen3-1.7B) to 11.3 nats.

The six remaining failures:

• Ezekiel's departing glory (the model picks angel).
• Free vs slave in Galatians 4.
• "Before Abraham was, I am" (the model picks was, the locally fluent tense).
• Son of god vs man. The target is lowercase because the eval was written for an uncased encoder.
• Not the righteous but sinners.
• Christ the end of the law.

The canonical-knowledge cases quote well-known verses, and Bible text is in the CPT data, so read 137/137 as in-distribution recall, not generalization.

Source-grounding ruler (701 cases)

Each case is a source context, a question, and a source-supported answer. At one point in the answer the model is offered the faithful sentence and a minimally edited foil. For the source-sensitive probes, a case is won when the context raises the faithful sentence's probability more than the foil's (pointwise mutual information). The cases are private. The method is described in EVAL.md.

The previous base was CPT'd on the older, ungrounded synthetic corpus. Pooled over the seven probes' 602 paired cases, the difference is +45 / −14 discordant cases (exact McNemar p < 0.0001). The other 99 cases are an eighth probe, invention, near the ceiling for every model and left out of the macro (see EVAL.md). Misattribution is the weakest probe: the model still sometimes puts a claim in the wrong speaker's mouth.

General English and unseen Bible text

C4 validation perplexity (65k tokens) went from 41.09 for the raw transplant to 18.15 after CPT. It is not directly comparable to the donor's perplexity, because the vocabularies differ.

A pre-registered base suite (test split, frozen gates) on held-out synthetic devotional text:

The one missed gate, agreement, comes from a tokenization artifact of that suite: every prefix ends in a bare space. Re-scored with natural word boundaries, agreement is 100%. The theology-text loss is perplexity on sermonic devotional prose. A base trained toward source-grounded writing is expected to score somewhat worse on it, so it is not a Bible-knowledge measure.

Quick Start — Text Continuation

pip install -r requirements.txt   # torch>=2.5, transformers>=5.0

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "toranb/theo-qwen-base"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32   # fp32 on CPU

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=dtype)
model.to(device).eval()

inputs = tokenizer("The parable of the prodigal son shows", return_tensors="pt").to(device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
# → (greedy, bf16 on GPU; fp32 on CPU words it slightly differently)
#   the depth of God’s love for us. It is a story of a son who has wasted his inheritance,
#   returned to his father, and was received with joy. The father’s response is a picture of
#   God’s love for us. The parable of the prodigal son is found in Luke

The tokenizer adds <bos> itself, matching the CPT framing. Generation stops at <eos>. Pass trust_remote_code=True to AutoTokenizer too. The tokenizer is standard, but AutoTokenizer reads config.json, and without the flag it stops to ask whether to run this repo's code.

Quick Start — Scoring Sentences

import torch
import torch.nn.functional as F

def sentence_logprob(text):
    ids = tokenizer(text).input_ids + [tokenizer.eos_token_id]   # <bos> text <eos>, as in CPT
    ids = torch.tensor([ids], device=model.device)
    with torch.no_grad():
        logits = model(ids).logits
    return F.log_softmax(logits[0, :-1], -1).gather(1, ids[0, 1:, None]).sum().item()

for word in ["grace", "law"]:
    print(word, sentence_logprob(f"Paul teaches that believers are justified freely by God's {word}, not by works of the law."))

scripts/theology_eval.py runs the full 545-case ruler this way.

Samples

Greedy decoding, 48 new tokens, from the parity check (torch 2.13.0, bf16 on an RTX 4090; other torch versions may diverge after a few dozen tokens, see Release Details):

When Jesus calls himself the good shepherd, he means that he is the one who knows his sheep, protects them, and lays down his life for them. He is the one who knows his sheep, and he knows them better than anyone else. He is the one who protects them, and

Grace is not a reward for good behavior. It is a gift from God. Grace is a gift from God. It is not a reward for good behavior. Grace is a gift from God. It is not a reward for good behavior. It is a gift from God.

The capital of France is a city of culture, history and art. Paris is the most visited city in the world, and it is also the most beautiful. The city is a place of museums, galleries, and monuments, and it is also a place of shopping

The second sample shows the repetition you should expect from greedy decoding on a base model. Use sampling, or a repetition penalty, for longer text.

Tokenizer

A case-preserving, NFC-normalized byte-level BPE with 32,768 tokens, trained on a sample of Bible-domain and general English text. It has four special tokens, <pad>=0, <bos>=1, <eos>=2 and <doc>=3, and twenty reserved tokens [unused1]–[unused20] at ids 32768–32787. The reserved tokens are ordinary (non-special) atomic tokens. This base model never saw them in training. The instruct model uses several as mode markers.

The only change from the training tokenizer is a post-processor that prepends <bos>. Token ids are unchanged.

Architecture

Qwen3-1.7B geometry with two changes:

• 28 decoder layers, hidden size 2048, SwiGLU MLP 6144
• 16 query heads, 8 key/value heads (GQA), head dim 128
• Q/K RMSNorm per head before RoPE; RoPE θ = 1,000,000
• pre-norm RMSNorm (ε 1e-6); no biases; tied input/output embeddings
• vocabulary 32,788 (Qwen3: 151,936)
• value-embedding residual on layers 0, 2, …, 20. The token embedding is projected to the KV width (value_embedding_proj), scaled per KV head (value_embedding_head_gate) and by a learned scalar (value_embedding_alpha), and added to the attention values. Each adds 2,097,161 parameters.
• max positions 2,048 (CPT ran at 1,024)

Parameter count: 1,499,628,643 (≈1.5B). Module names follow Qwen3 (model.layers.N.self_attn.q_proj, …), so PEFT target_modules written for Qwen3 work unchanged.

Repository Contents

Scripts

Limitations

• Base model. It continues text; it does not answer questions or follow instructions reliably. Use theo-qwen-instruct for that.
• Specialized for biblical and theological English. Its general-English and multilingual ability relative to Qwen3-1.7B has not been measured directly. Expect some loss from the parameter cut (1.7B → 1.5B), the much smaller vocabulary, and domain CPT.
• It reflects the theological traditions represented in its Christian devotional and theology-domain training material. Its preferences are distributional, not authoritative.
• It can misquote or misattribute Scripture, invent references, and blend passages. Check anything quoted before using it.
• English only. Trained at 1,024 tokens. The positional limit is 2,048. Longer inputs raise an error.
• The training data is not public, which limits external auditing. The theology ruler's canonical-knowledge cases overlap by design with Bible text in training.

Release Details

• Exported from cedar-v2-grounded-s20260924-v1/latest.pt (step 4,334; sha256 075eac91…). This is the exact checkpoint theo-qwen-instruct was trained from.
• The weights are bf16, bit-exact: the model was trained and stored in bf16, so this release loses nothing. fp16 is not recommended, because some value-embedding scalars (alpha as small as 1.45e-5) fall into fp16's subnormal range.
• Loading requires trust_remote_code=True.
• The HF implementation reproduces the training code's logits bit for bit in bf16 on GPU (torch 2.13.0, transformers 5.15.1), with and without padding masks, and its greedy generations token for token.
• Minimum supported: torch 2.5 (the attention call uses SDPA's enable_gqa, added in 2.5) with transformers 5.0. Smoke-tested at exactly torch 2.5.1 / transformers 5.0.0.
• Tested in a clean install of torch 2.14.1 and transformers 5.20.2. Logits match the training stack to 2.5e-5 in fp32, the same code path. In bf16, newer torch kernels round differently (up to 0.3 in a logit), which can change a long greedy continuation or a near-tied eval case: 540/545 instead of 539/545.
• RoPE inverse frequencies are rounded to bf16 (rope_inv_freq_dtype), because that is what the weights were trained with. Setting it to "float32" changes C4 loss by less than 0.001 nats.
• config.json declares dtype: bfloat16. On CPU, load with dtype=torch.float32 for full-precision math.

Attribution

Built on Qwen3-1.7B by the Qwen team, Alibaba Cloud, released under the Apache License 2.0. Its transformer layers are carried over here, modified by LoRA merge, vocabulary transplant and continued pretraining. Released under Apache 2.0; see LICENSE. The Apache 2.0 license covers the model weights and code; it grants no rights in any third-party text the model may reproduce.

Citation

@misc{theoqwenbase2026,
  author       = {Toran Billups},
  title        = {Theo Qwen Base: A Grounded Bible-Domain Transplant of Qwen3-1.7B},
  year         = 2026,
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/toranb/theo-qwen-base}
}