Rudushi-1 (1.5B)

Rudushi-1 is the first open, Bangla-first instruction model from the Rudushi initiative. It is a QLoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct trained to answer with structured steps (ধাপ ১ / ধাপ ২ / চূড়ান্ত উত্তর) and to check its own work (খসড়া → যাচাই → সংশোধন).

This repository contains merged fp16 full weights.

⚠️ Research checkpoint. This is a 1.5B model trained on a single T4. It is useful for Bangla instruction following and for demonstrating structured reasoning, but it is not a general reasoner and it can be confidently wrong. Verify facts, numbers and dates.

Table of Contents

1. Model Summary
2. Quickstart
3. Answer modes
4. Training
5. Evaluation
6. Limitations
7. বাংলায় সংক্ষেপে
8. Jessore dialect
9. License
10. Citation

Model Summary

• Type: causal decoder-only language model (instruction-tuned)
• Architecture: Qwen2.5 — RoPE, SwiGLU, RMSNorm, grouped-query attention (12 query / 2 KV heads), 28 layers
• Parameters: 1.54B total (1.31B non-embedding)
• Base context length: 32,768 tokens (this fine-tune was trained at 512 tokens)
• Languages: Bengali (primary), English
• Release format: sharded safetensors (1 GB shards), fp16
• Fine-tuning: 4-bit NF4 QLoRA, fp16 compute, assistant-only loss

What it is trained to do

Quickstart

pip install "transformers>=4.45" torch accelerate safetensors

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "SheikhMinds/rudushi-1-1.5b"
model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto", torch_dtype="auto")
tok = AutoTokenizer.from_pretrained(repo)

messages = [
    {"role": "user", "content": "একটি দোকানে ৩টি কলমের দাম ২১ টাকা। ৪টির দাম কত? ধাপে ধাপে উত্তর দাও।"}
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=False,
    pad_token_id=tok.pad_token_id,
    eos_token_id=tok.eos_token_id,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Expected shape (not guaranteed):

ধাপ ১: একটির দাম = ২১ ÷ ৩ = ৭ টাকা।
ধাপ ২: ৪টির দাম = ৭ × ৪ = ২৮ টাকা।
চূড়ান্ত উত্তর: ২৮ টাকা।

The snippet above loads full weights. If this repository is an adapter-only release, load Qwen/Qwen2.5-1.5B-Instruct first and wrap it with PeftModel.from_pretrained(model, "SheikhMinds/rudushi-1-1.5b").

Answer modes

The demo Space exposes three modes, and you can reproduce them in code:

The self-check mode is inference-time reflection: the weights do not change, the model simply gets a second pass over its own draft. It sometimes helps and sometimes hurts — both numbers are reported below.

Training

Data

SheikhMinds/rudushi-bangla-sft contains the published artifacts. The full training mix is built deterministically by the Rudushi data pipeline:

Reasoning generators cover: unit price, discount, linear equations, averages, ratios, unit conversion, arithmetic sequences, work-rate, syllogisms, and verification traces.

Recipe

Framework versions used for this run:

Reproduce with the Rudushi repository:

make reasoning-data
make reasoning-train
make reasoning-eval
make base-eval
make merge

Evaluation

Loss is not capability. We report exact-match accuracy on held-out synthetic problems (generated with a different seed than training) and the change caused by reflection.

Held-out synthetic problems, generator seed 777 (unseen in training).

By category

These are bounded, synthetic distributions. High accuracy means the model learned this format and these operations; it does not establish general mathematical reasoning. A base-model column is included so the fine-tune can be judged against its starting point.

Limitations

• Narrow reasoning data. Arithmetic/logic coverage is synthetic and finite; expect failures outside those patterns.
• Small model. 1.5B parameters cannot match 7B+ models on knowledge or multi-hop tasks.
• Hallucination. Names, numbers, dates and citations can be fabricated.
• No RLHF/DPO. Helpfulness and refusal behaviour are not preference-tuned.
• Dialect coverage. Standard Bangla dominates. Jessore dialect is a separate, untrained run.
• Long context. The base supports 32k tokens, but this fine-tune was trained at 512; long-context quality is not evaluated.
• Not for safety-critical use. No medical, legal, financial or safety advice.

বাংলায় সংক্ষেপে

রুদুষি-১ একটি বাংলা-প্রথম ১.৫বি মডেল। এটি ধাপে ধাপে সমাধান ও নিজের উত্তর যাচাই করার ফরম্যাট শেখে। এটি সাধারণ বুদ্ধিমত্তার মডেল নয় — ছোট, সীমিত ডেটায় প্রশিক্ষিত একটি গবেষণামূলক মডেল। গুরুত্বপূর্ণ তথ্য সবসময় যাচাই করুন।

যশোর অঞ্চলের উপভাষার জন্য আলাদা ডেটা ও আলাদা রান দরকার; এই চেকপয়েন্টটি উপভাষায় বিশেষভাবে প্রশিক্ষিত নয়। বিস্তারিত: Rudushi রিপোর docs/dialect-jessore.md।

Jessore dialect

The Bangla spoken in Jessore (যশোর) is a clear, melodic form of the southwestern Bangla dialect group, described as shaped by both Rarhi and Bangali strands. It is a living regional identity, not "incorrect" Bangla.

Example pairs from the draft seed:

The design goal is optional, context-aware dialect use: if a user writes standard Bangla, the answer should stay standard.

License

• Model weights and training code: Apache-2.0
• Base model: Qwen/Qwen2.5-1.5B-Instruct (Apache-2.0)
• Data: each upstream source keeps its own licence; see the dataset card.

Citation

@misc{rudushi1,
  title        = {Rudushi-1: An Open Bangla-First Instruction Model with Structured Reasoning},
  author       = {Rudushi},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/SheikhMinds/rudushi-1-1.5b}},
  note         = {QLoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct}
}

Model card contact

Open an issue or discussion on the Rudushi Space.