FreeSolo-L-341M
Paper · GitHub · Models · Usage
RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation.
FS-L retains a bounded pool of chunks by key norm and recency, then reads four selected chunks. Its attention scales linearly with sequence length. This 341M research base model was trained on FineWeb-Edu at a 4,096-token context.
Try it
hf download zekun-shi/FreeSolo-L-341M freesolo-source.zip --local-dir ./FreeSolo-L-341M
python -m pip install ./FreeSolo-L-341M/freesolo-source.zip
freesolo-generate --model zekun-shi/FreeSolo-L-341M --output generation.json
The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command.
The idea
FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys.
Results
| Context | 2k | 4k | 8k | 16k |
|---|---|---|---|---|
| RULER mean | 32.32 | 21.72 | 15.27 | 10.36 |
12 tasks at 2k/4k; 13 at 8k/16k, where QA-2 is available. 100 examples per task, seed 42. These are checkpoint evaluation results; the figures show the paper's model-family comparison.
This is a base model without instruction tuning. Retrieval is task dependent, especially when several facts or values must be combined. Training details, artifact checksums, and citation are in Usage.