FreeSolo-B-743M
Paper · GitHub · Models · Usage
RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation.
FS-B searches full-prefix and norm-filtered forests of chunk mean keys, reranks their candidates, and reads four chunks. Its attention is superlinear and subquadratic. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context.
Try it
hf download zekun-shi/FreeSolo-B-743M freesolo-source.zip --local-dir ./FreeSolo-B-743M
python -m pip install ./FreeSolo-B-743M/freesolo-source.zip
freesolo-generate --model zekun-shi/FreeSolo-B-743M --output generation.json
The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command.
The idea
FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys.
Results
| Context | 2k | 4k | 8k | 16k |
|---|---|---|---|---|
| RULER mean | 41.18 | 36.88 | 33.02 | 31.17 |
Common 12-task set at every length, excluding QA-2. 100 examples per task, seed 42. These are checkpoint evaluation results; the figures show the paper's model-family comparison.
This is a base model without instruction tuning. Retrieval is task dependent, especially when several facts or values must be combined. Training details, artifact checksums, and citation are in Usage.