Pragya · EXP3 Chat

A 28.5M-parameter chat model, fine-tuned from Pragya EXP2. For testing, not for production.

⚠️ Not a product

This is a small experimental chat model. It is not aligned, factually reliable, or safe. Its replies are short and often wrong.

---

Summary

---

What changed from EXP2

EXP2 is a base model: it continues text but has no way to signal the end of a reply. EXP3 Chat is fine-tuned on single-turn conversations, with each reply closed by the marker *end*. The model learns the marker during fine-tuning, so replies can be cut at it.

The architecture, tokenizer and weights are unchanged, apart from the fine-tuning updates.

---

Fine-tuning

Training used batch size 1 because the model rejects padded batches. Gradient checkpointing is off, as the model does not support it.

---

Results

BananaMind Base Bench 1.1 (official runner)

Overall Elo 976. Raw accuracy 174/350 (49.71%), weighted accuracy 44.89%, against 25% chance.

Context tracking and world knowledge improved. Code completion dropped by 22 points, which fits chat data containing little code.

lm-evaluation-harness (0-shot, acc_norm)

Intelligence Index: 14.21 (base 12.41), using the same formula as before:

N(s, c) = 100 × (s − c) ÷ (100 − c)
Index = (N(HellaSwag, 25) + N(ARC, 25) + N(PIQA, 50) + 0.65 × N(ArithMark-3, 25)) ÷ 3.65

The ARC-Easy gain drives most of the index change. Treat the index change as suggestive until the embedding check above is resolved.

AxiomicLabs reasoning benchmarks

Report acc_norm for Tiny Theory of Mind, since its card specifies length-normalised scoring.

---

Quick Architecture

## Quick start

pip install torch transformers safetensors xformers

The model loads through transformers with remote code. Generate one prompt at a time, since padded batches are not supported.

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "RuVI-AI/Pragya-EXP3-Chat"   # or a local folder
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).to("cuda:0").eval()

def chat(question, max_new_tokens=200):
    prompt = f"User: {question.strip()}\nAssistant:"
    enc = tok(prompt, return_tensors="pt").to("cuda:0")
    out = model.generate(
        **enc,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.7,
        top_k=12,
        top_p=0.95,
        repetition_penalty=1.3,
        no_repeat_ngram_size=3,
        stop_strings=["*end*", "User:"],
        tokenizer=tok,
        pad_token_id=tok.pad_token_id,
    )
    text = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
    return text.split("*end*")[0].split("User:")[0].strip()

print(chat("What is Internet?"))

Replies end at *end* when the model writes it. If the model doesn't write the marker, generation stops at max_new_tokens and the reply is cut at the first User:.

---

Sampling

These settings gave fluent replies in testing:

temperature        0.7
top_k              12
top_p              0.95
repetition_penalty 1.3
no_repeat_ngram    3

Greedy decoding repeats phrases, so use sampling.

---

Capabilities and limitations

It can: answer simple single-turn questions briefly, and close most replies with *end*.

It cannot: reliably answer factual questions, handle multi-turn conversations, write useful code, use context longer than 1024 tokens, or handle non-English input.

• Small scale. Replies are short and often factually wrong.
• Single-turn training only. Multi-turn context is not trained.
• No safety filtering. Replies can be biased, false or toxic.
• EOS is not learned. The end-of-sequence token is still unused. The *end* marker is the stop signal.

---

License

Apache 2.0. The fine-tuning data (HuggingFaceTB/smol-smoltalk) and the base model's training data carry their own licences. Check upstream terms before redistributing derivatives.

---

Citation

@misc{pragya-exp3-chat,
  title  = {Pragya EXP3 Chat: a 28.5M chat model fine-tuned from Pragya EXP2},
  author = {Arush},
  year   = {2026},
  note   = {Experimental checkpoint, not for production use},
  howpublished = {\url{https://huggingface.co/RuVI-AI/Pragya-EXP3-Chat}}
}