GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)

An embedding-only Geometric Rotation Transformer (GRTModel) contrastively fine-tuned from MatterWaveAI/grt-30m for semantic textual similarity and dense retrieval using planar-pair circular pooling (sentence-transformers).

Instead of dense linear projections (nn.Linear), GRT represents token embeddings in polar coordinates (phase_angles + log_magnitude) and transforms hidden states through cascades of content-modulated 2×2 Givens rotations on SO(D) interleaved with butterfly permutations. Because every hidden state is structured as D / 2 = 128 planar (x, y) coordinate pairs, circular pooling decomposes each pair into its unit direction (cos θ, sin θ) and radius r, averages the unit directions and radii across non-padding tokens, and recombines them into a 256-dimensional sentence embedding.

---

Model Details

---

Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

Starting from the pretrained MatterWaveAI/grt-30m causal checkpoint (lm_head.output_scale stripped to form a headless GRTModel encoder), the 151,669 lexical embedding.phase_angles (19.41M parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the 9,740,196 rotational backbone parameters + 151,669 token log_magnitude weights (9,891,865 trainable parameters) in FP32 on an Intel Arc 140T GPU (xpu:0, 32 GB) using fused oneAPI DPC++ SYCL Givens and Clifford kernels (~75s total wall-clock time):

---

Evaluation Results & 17-Way Classical Geometric, Symplectic & Toroidal Optimizer Ablation (STS-B)

Evaluated on the STS Benchmark (sentence-transformers/stsb) validation and test splits using circular pooling and cosine similarity across a 17-way controlled optimizer ablation on the identical 508-step schedule (150 steps AllNLI + 358 steps STS-B, seed = 42):

Memory, Speed & Regime Tradeoffs Across GRT Optimizers (optimization_grt.py)

For the 9,891,865 trainable parameters (39.57 MB in FP32: 190,464 rotation angles across 84 cascade tensors, 9,175,040 Cl(2,0) coupling parameters, and 526,361 radial scale / norm / log_magnitude parameters):

---

Usage

1. sentence-transformers (Recommended)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")

sentences = [
    "Plants convert sunlight into chemical energy through photosynthesis.",
    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
    "The stock market experienced volatility during the afternoon session.",
]

embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities)

2. Hugging Face transformers (GRTModel + Circular Pooling)

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-embedding"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()


def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
    radii = pairs.norm(dim=-1, keepdim=True)
    directions = pairs / torch.clamp(radii, min=1e-9)
    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)


inputs = tokenizer(
    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**inputs)
    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])

print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
8. Rotational Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-tts
9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}