AETHER-7B-7Attn-base

Which Aether model should I use?

Model	What it is	Pick it if
Aether-7B-5Attn	6.59B MoE base. Fully open - weights + data recipe + training code + all 162k-step logs + checkpoints	You want to audit, verify or rebuild a foundation model end to end
Aether-7B-5Attn-it	The same model, instruction-tuned	You want it to answer rather than continue text
AETHER-7B-7Attn-base	Same 49-layer architecture, a different checkpoint. Open weights	You want a second run of this architecture to compare against
Aether-6B-11Attn-base	121 layers, 11 sequence-mixing mechanisms in one network - attention, Mamba-2, Hyena, GDN, MLA - on an 11x11 Latin square	You research heterogeneous sequence mixing. It is a mid-training research artifact

All four load the same way:

AutoModelForCausalLM.from_pretrained(MODEL, trust_remote_code=True, dtype=torch.bfloat16)

Open-weight release — weights and modeling code. Training data, recipe and logs are not included.

To our knowledge, the first foundation model to implement seven heterogeneous attention mechanisms within a single architecture.

Attention layout

Seven distinct attention mechanisms (A–G) are distributed across 49 layers in a 7×7 Latin square — every mechanism appears exactly once in each row and each column, so no single mechanism dominates any depth band.

The layer-to-mechanism assignment is defined by LATIN_SQUARE_7x7 in modeling_aether_v2_7way.py.

Model


Parameters	6.59B total / ~3B active (Mixture-of-Experts)
Type	Decoder-only language model, heterogeneous multi-attention architecture
Precision	bfloat16
Languages	Korean, English
Stage	Pretrained base (no instruction tuning)
Developer	VIDRAFT

Unlike conventional models that repeat a single attention mechanism across all layers, AETHER-7B-7Attn composes seven distinct attention mechanisms inside one network. This heterogeneous design is the defining characteristic of the AETHER architecture.

Release scope

This repository ships the weights and the modeling code needed to load them.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "FINAL-Bench/AETHER-7B-7Attn-base"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()

trust_remote_code=True is required — aether_v2_7way is a custom architecture.

Not released here: the training data, data mixture, training code, logs and hyperparameters. Those are published only for the fully-open sibling Aether-7B-5Attn, which shares this architecture. This model is therefore open-weight, not open-source, in the sense that it cannot be reproduced from what is in this repository.

Notes for use

Run at batch_size=1. The NSA branches do not consume a padding mask, so batching padded sequences can silently corrupt results.
No KV cache. use_cache is already False in the config; generation re-runs the full forward pass per token and is correspondingly slow.
Base model, no safety alignment. It continues text rather than following instructions. Do not deploy it as-is.

Intended use

Released for research, evaluation and benchmarking of the released weights. This is a base (pretrained) model — it continues text rather than following instructions.

License

Apache-2.0 — the released weights are free to use, modify and redistribute under the terms of the Apache License 2.0.

The architecture implementation is included in this repository under the same licence. The training pipeline and data recipe are not part of this release.

VIDRAFT · Sovereign AI