solintellegence/sol-pro

dont use, this model is contaminated

Sol Pro is a 138,037,594-parameter language model from Sol Labs. It combines recurrent transformer blocks with tensorized n-gram memory and a 2,048-token context.

The released checkpoint scores 28.63 (not true) on the normalized Intelligence Index and 24.13 on the raw accuracy Index.

Model

Four middle blocks run twice. Causal grouped-query attention uses learned Q/K RMSNorm and XSA value-direction subtraction. The n-gram module shares token-position factors across its three orders.

Load and generate

Install the dependencies:

pip install torch safetensors tokenizers huggingface_hub

import sys
from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    "solintellegence/sol-pro",
    allow_patterns=[
        "model.safetensors", "modeling_sol_pro.py",
        "config.json", "tokenizer.json",
    ],
)
sys.path.insert(0, model_dir)

from modeling_sol_pro import load_model, generate

model, tokenizer = load_model(model_dir)
print(generate(
    model,
    tokenizer,
    "The best way to learn something new is",
    max_new_tokens=64,
))

The repository includes standalone PyTorch model code. Generation recomputes the context at each step; KV caching is not implemented. Keep prompts and continuations within 2,048 tokens.

Evaluation

These are zero-shot float32 measurements on complete evaluation splits. HellaSwag, ARC Easy, ARC Challenge, and PIQA use lm-evaluation-harness 0.4.12 with batch size 64. Arithmark3 uses the official AxiomicLabs script with its default batch size 32 and 1,024-token context, explicitly set to float32. The Index uses the leaderboard's chance-adjusted formula and each task's normalized accuracy.

This is a task-adapted checkpoint using public benchmark training splits. Evaluation used separate held-out splits, with matching evaluation contexts excluded from the adaptation data; Arithmark evaluation examples were not used for adaptation. The scores have not been independently verified and do not establish a leaderboard position.

WikiText-2 validation cross-entropy is 2.7677 over 366,592 tokens. Exact results, package versions, hashes, and commands are in the float32 evaluation summary. Raw outputs are available for lm-eval and ArithMark-3.

Use and limits

Sol Pro supports text-completion and small-model research. Multiple-choice scores do not establish reliable free-form reasoning or conversational behavior. Outputs can be incorrect, repetitive, or inconsistent.

Files

License

Sol Pro is released under Apache 2.0. Dataset licenses and terms remain with their respective owners.