FreeSolo-B-341M

RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation.

FS-B searches full-prefix and norm-filtered forests of chunk mean keys, reranks their candidates, and reads four chunks. Its attention is superlinear and subquadratic. This 341M research base model was trained on FineWeb-Edu at a 4,096-token context.

Try it

hf download zekun-shi/FreeSolo-B-341M freesolo-source.zip --local-dir ./FreeSolo-B-341M
python -m pip install ./FreeSolo-B-341M/freesolo-source.zip
freesolo-generate --model zekun-shi/FreeSolo-B-341M --output generation.json

The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command.

The idea

FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys.

Results

This is the 341M checkpoint used for the paper's FS-B speed measurements. See speed replication for the complete protocol and expected throughput.

This is a base model without instruction tuning. Retrieval is task dependent, especially when several facts or values must be combined. Training details, artifact checksums, and citation are in Usage.