GRT-30M (24-Hour Qwen 3 Distillation Preview)

Multi-Round Curriculum Checkpoint Available: For the multi-round HuygensSymplecticAdam curriculum adaptation of GRT-30M (48.07% 9-task SLM-Bench multiple-choice macro average, 99.20% GSM8K 4-way verification accuracy, 7.60 GSM8K perplexity), see MatterWaveAI/grt-30m-curriculum.

A Geometric Rotation Transformer (GRTForCausalLM) distilled from Qwen 3 (sharing the 151,669-token Qwen 3 BPE vocabulary) in a single 24-hour distillation run (10.79B tokens at ~123.9k tok/s across 8x NVIDIA A100-SXM4-80GB). GRT replaces every dense linear projection (nn.Linear) in attention (Q, K, V, O) and feed-forward (expand, contract) with cascades of content-modulated 2×2 Givens rotations on SO(D) interleaved with butterfly permutations.

This is a base research checkpoint distilled from Qwen 3, not an instruction-tuned assistant. It demonstrates that a rotation-only student backbone with just 9.74M non-embedding parameters (29.31M total stored parameters with the 151,669-token Qwen 3 vocabulary):

1. Distills stably from Qwen 3 at 123,900 tok/s to val_loss = 4.4776 (val_ppl = 88.02, 27.2% top-1 token accuracy over a 151,669-token vocabulary).
2. Achieves bit-for-bit 4-way greedy generation parity across PyTorch ROCm (gfx1151), llama.cpp CPU (AVX2), llama.cpp Vulkan (RADV), and llama.cpp ROCm (HIP).
3. Runs at 3,790 tok/s prefill (Vulkan) and 87.4 tok/s decode (CPU, 16 threads) in pure FP32 on an AMD Ryzen AI Max+ 395 (Strix Halo / gfx1151), and 5,760 tok/s forward in BF16 with the fused GRT HIP kernels.
4. Beats random chance across 6 of 7 zero-shot multiple-choice benchmarks (ARC-Easy 29.50%, HellaSwag 29.20%, OpenBookQA 28.40%, CommonsenseQA 21.70%, BoolQ 51.53%, QNLI 51.50%), matching dense models (Pythia-14M / Pythia-31M) of comparable backbone capacity.

---

Model Details

Because grt-30m shares the full 151,669-token Qwen 3 vocabulary at hidden_size = 256 for direct token-aligned distillation from Qwen 3, the tied polar embedding table (phase_angles [151669, 128] + log_magnitude [151669]) accounts for 19.57M parameters (66.8% of stored weights), leaving 9.74M stored parameters for the 12-layer rotational transformer backbone.

---

24-Hour Qwen 3 Distillation Summary

---

4-Backend Exact Greedy Generation Parity

model.safetensors (transformers) and grt-30m-f32.gguf (llama.cpp) produce identical token-for-token greedy continuations (temperature = 0, 16 new tokens) across all four supported backends on AMD Strix Halo (gfx1151):

• PyTorch ROCm (gfx1151, fused HIP kernels)
• llama.cpp CPU (x86_64 AVX2)
• llama.cpp Vulkan (RADV gfx1151)
• llama.cpp ROCm (HIP gfx1151)

---

Benchmarks

1. Zero-Shot Multiple-Choice & Held-Out Perplexity

Evaluated zero-shot (no task fine-tuning) via teacher-forced length-normalized log-likelihood scoring on checkpoint-20000:

SLM-Bench Task-Adapted Curriculum Benchmarks (grt-slm-bench / training/slm_bench.py)

Evaluated across the 10-task SLM-Bench core suite on an AMD Ryzen AI Max+ 395 (Radeon 8060S / ROCm gfx1151), comparing the zero-shot distilled checkpoint against task fine-tuning with PairAdam, SymplecticAdam, and HuygensSymplecticAdam:

GSM8K Held-Out Reasoning Perplexity (openai/gsm8k, test split, n = 500)

2. Inference & Kernel Throughput on AMD Strix Halo (Radeon 8060S / gfx1151)

llama.cpp (grt-30m-f32.gguf, FP32)

PyTorch ROCm (gfx1151, BF16, Fused HIP Kernels)

3. Circular-Pooled Sentence Encoder (MatterWaveAI/grt-30m-embedding)

We also release MatterWaveAI/grt-30m-embedding (GRTModel + SentenceTransformer), adapted from grt-30m using planar-pair circular pooling (pooling_mode="circular", averaging unit directions and pair radii separately across 128 planar (x, y) pairs). On an Intel Arc 140T GPU (xpu:0, fused oneAPI SYCL kernels), the 151,669 lexical embedding.phase_angles were frozen while training the 9,891,865 rotational backbone and log_magnitude parameters with PairAdam in two stages:

¹ Footnote: Under the identical 508-step schedule and seed (42), standard AdamW (1× coupling LR, full diagonal second moments) reaches 72.89 Val / 63.17 Test Spearman vs. 71.79 / 62.42 for PairAdam (3× coupling LR, factored second moments), suggesting PairAdam's 3× coupling multiplier is primarily beneficial for large-batch pretraining rather than short small-batch (B = 32) fine-tuning.

---

Usage

PyTorch (transformers)

import torch
from transformers import AutoTokenizer, GRTForCausalLM

# Optional: register fused CUDA/ROCm Givens kernels if grt-kernels is installed
try:
    from grt_kernels.torch_ops import ensure_grt_ops
    ensure_grt_ops()
except ImportError:
    pass

model_id = "MatterWaveAI/grt-30m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GRTForCausalLM.from_pretrained(model_id, dtype=torch.float32).eval()

prompt = "The history of the city"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

llama.cpp (GGUF)

./build/bin/llama-completion \
  -m grt-30m-f32.gguf \
  -p "Once upon a time" \
  -n 32 --temp 0 --seed 0

Verify file integrity against SHA256SUMS:

sha256sum --ignore-missing -c SHA256SUMS

---

Architecture Overview

GRT replaces dense weight matrices (nn.Linear) in both attention and FFN blocks with cascades of learned 2×2 Givens rotations arranged in a butterfly topology. At each stage k, pairs of coordinates (x0, x1) are rotated by an angle theta_k modulated by 4-component Cl(2,0) multivector features:

feat(x0, x1) = [x0^2 + x1^2,  x0,  x1,  x0 * x1_next - x1 * x0_next]
theta_k      = alpha_k + tanh(norm(feat(x0, x1)) @ C_k)
r0           = x0 * cos(theta_k) - x1 * sin(theta_k)
r1           = x0 * sin(theta_k) + x1 * cos(theta_k)

• GRTEmbedding & GRTCosineHead — Token embeddings are stored in polar form (phase_angles + log_magnitude) and tied to a scaled cosine-similarity language modeling head (lm_head.output_scale = 40.23).
• GRTAttention — 11-stage content-modulated Givens butterfly cascades for Q, K, V, O, paired with standard scaled dot-product attention and learnable RoPE frequencies.
• GRTMixer & GRTFFN — Cross-head Givens mixing stage followed by 21-stage rotational FFN expansion/contraction and phase-preserving GRTRadialGELU.
• GRTPearlNorm — Pair-aware RMS normalization that normalizes pair radii sqrt(x_{2i}^2 + x_{2i+1}^2) while preserving phase angles.

---

Intended Use & Limitations

• Research preview. Intended for studying geometric/rotational neural architectures, kernel benchmarking, and multi-backend (PyTorch / llama.cpp / vLLM) verification.
• Base causal LM only. Not instruction-tuned, no chat template, no safety filtering, and prone to repetition under greedy decoding at small parameter scale.

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
8. Rotational Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-tts
9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}