Archaea-74M

Archaea-74M is a decoder-only causal language model with approximately 74 million parameters, pretrained from scratch on BetterDataset-2M. The model uses a LLaMA-style architecture with Grouped Query Attention (GQA) and was trained using BF16 mixed precision.

This release represents approximately 1.23 billion trained tokens out of a planned 1.6 billion token pretraining run, making it a substantial intermediate checkpoint that captures most of the intended training curriculum while leaving room for future scaling and refinement.

---

Model Card

---

Architecture graph

Architecture

Transformer Configuration

The model implements Grouped Query Attention, reducing KV-cache memory requirements while maintaining strong representational capacity for a model of this scale.

---

Training

Dataset

Archaea-74M was pretrained on GODELEV/BetterDataset-2M, a multi-source corpus composed of:

• General web text
• Conversational content
• Knowledge-focused material
• Educational content
• Instruction-like examples
• Technical and programming text

The complete corpus contains approximately 1.6 billion tokens.

Training Progress

Optimization

Training Statistics

Training Loss Curve

Learning Rate Schedule

---

Evaluation

Evaluated using EleutherAI LM Evaluation Harness.

Benchmark Results

Done on 0-Shot

Evaluation Notes

Failed tasks from previous runs:

• SocialIQA: dataset loading incompatibility.
• Arithmetic benchmark: task identifier changes in recent lm-eval versions.

---

Sample Generations

The following examples were generated directly from the pretrained base model.

Example 1

Input

Once upon a time

Output

Once upon a time

In the days after, I was told that I could not make a living. So I said to myself,
“I want to go back to my family.” I said, “This is a good one. You can take me to the doctor
or something and do it with you.”

The doctor asked me how I felt about this. He said he did. But I said,
“You can't have a baby.”

Example 2

Input

The horizon looked

Output

the horizon looked like a big black hole. The sun was shining and the wind was blowing.

The sun was shining with the sun, but it was still there. The stars were on top of each other.

The sun was falling as a result of the weather. The sun was going up.

Example 3

Input

what is the future of ai

Output

what is the future of Ai?

I don't know what you're talking about, but I'm sure that's a good reason for you to be optimistic.

It doesn't matter if it doesn't take away your wandering in a positive direction.

The answer is simple: I'll leave you with a feeling of uncertainty and optimism.

These outputs illustrate the model's ability to continue prompts coherently while also demonstrating typical limitations of small-scale pretrained language models, including repetition, topic drift, and inconsistent factual reasoning.

---

Usage

Installation

pip install torch transformers

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "GODELEV/Archaea-74M"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

Text Generation

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "GODELEV/Archaea-74M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "The future of artificial intelligence"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.8,
        do_sample=True,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

---

Repository Structure

Archaea-74M/
├── config.json
├── generation_config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── Archaea74M_Training_Loss_Curve.png
├── Archaea74M_Learning_Rate_Schedule.png
└── README.md

---

Limitations

Archaea-74M is a base pretrained model and has not undergone:

• Instruction tuning
• RLHF
• Preference optimization
• Safety alignment

Known limitations:

• Hallucinations and factual inaccuracies
• Limited reasoning due to model scale
• Sensitivity to prompt phrasing
• Fixed 1024-token context window
• Not suitable for high-stakes applications

---

Future Work

• Instruction tuning
• Expanded benchmark coverage
• Longer context lengths
• Improved data quality and curriculum design

---

Citation

@misc{archaea74m,
  title={Archaea-74M},
  author={Akshit Kumar},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/GODELEV/Archaea-74M}
}