ruGPT-3 XL 8k

A 1.3B-parameter GPT-2-style language model for Russian with an extended context window of 8192 tokens, trained via continued pretraining from evilfreelancer/ruGPT3XL.

This is a base (pretrained) model, not instruction-tuned.

Model Details

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "evilfreelancer/ruGPT3XL-8k"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)

inputs = tokenizer("Москва - столица", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.2,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Context Extension: 2k -> 4k -> 8k

The original ruGPT3XL uses Learned Absolute Positional Embeddings (APE): the position table embed_positions is a plain nn.Embedding(max_position_embeddings, hidden_size) that is trained together with all other weights. This means the model has never seen position indices beyond 2047 and cannot generalize to longer sequences without fine-tuning.

Additionally, the model uses alternating sparse attention where the attention mask grid is built dynamically as num_blocks = max_position_embeddings // sparse_block_size, so increasing max_position_embeddings automatically adjusts the sparse grid without any architectural changes.

Strategy

Context was extended in two steps: 2k -> 4k, then 4k -> 8k, continuing from the previous checkpoint each time.

Step 1 - Positional embedding tiling. The existing embedding matrix is kept intact for known positions (0 to N-1). New positions are filled by cycling through the original table:

position 2048 <- weights of position 0
position 2049 <- weights of position 1
...
position 4095 <- weights of position 2047

position 4096 <- weights of position 0   (second cycle)
...
position 8191 <- weights of position 4095

This is deliberately chosen over linear interpolation: interpolation perturbs all existing embeddings and causes severe perplexity regression on short contexts. Tiling preserves exact weights for positions 0..N-1, so the model does not "forget" how to handle short sequences.

Step 2 - Mixed-length dataset. Training uses a 60/40 mix of long and short examples:

• Long (60%): multiple news articles from IlyaGusev/gazeta packed together with EOS tokens until reaching the target context length. All packed samples exceed half the target length, ensuring the model is consistently exposed to new position indices.
• Short (40%): single-article chunks up to half the target length. Prevents forgetting short-context behavior.

Step 3 - Continued pretraining. 3 epochs, lr=5e-6, cosine decay, warmup_steps=50, gradient_checkpointing=True, bfloat16, gradient_accumulation_steps=8, hardware: RTX 4090 (48 GB VRAM).

Note on OOM. Training at 8k context caused CUDA memory fragmentation during backpropagation, crashing at step 517/936 despite ~1 GB of technically free VRAM. Fix: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. After this, peak usage dropped from 46.8 GB to 38.5 GB and training completed without issues.

Perplexity

Evaluated on the test split of IlyaGusev/gazeta, strategy non_overlapping, bfloat16.

Regression on the original 2k context is +0.09 PPL - essentially unchanged. The 4k evaluation on the 8k model is slightly better than the intermediate 4k checkpoint (11.99 vs 12.04), indicating that continued pretraining improved overall quality.

VRAM Requirements (inference, batch=1, bfloat16)

Model weights occupy ~2.67 GiB (bfloat16). Overhead from KV cache and activations grows roughly linearly up to ~2k (sparse attention helps) and becomes near-quadratic beyond that. GPUs with 8 GB VRAM are practical up to ~3.5-4k context.

Generation Speed (bfloat16, 64 new tokens, batch=1, RTX 4090)

Speed is measured for autoregressive decoding with KV cache. The 2x step from 4k to 8k prompt length causes only ~1.8x slowdown (67 -> 38 tok/s), consistent with the linear scaling expected from sparse attention.

Sparse Attention

Inherited from the base model: even-numbered layers (0, 2, 4, ...) use block-sparse causal attention, odd-numbered layers use standard dense causal attention. The sparse pattern is computed from config.json at model init and does not require DeepSpeed at inference time.

Limitations

• Base model, not instruction-tuned. Works best for text completion.
• Primarily Russian text. Limited capability in other languages.
• Content may be biased, factually incorrect, or offensive - inherited from the original pretraining corpus.
• At 8k context, inference requires ~16 GB VRAM (bfloat16, batch=1).

Training Details

Citation

@misc{rugpt3xl_8k,
  title={ruGPT-3 XL 8k - extended context window via positional embedding tiling},
  author={Pavel Rykov},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/evilfreelancer/ruGPT3XL-8k}
}

Links

• evilfreelancer/ruGPT3XL - base model
• ai-forever/rugpt3xl - original Megatron-LM checkpoint
• IlyaGusev/gazeta - fine-tuning dataset
• Extending Input Contexts via Segmented Sequences (arXiv:2310.14633)
• Impact of Positional Encoding on Length Generalization (arXiv:2305.19466)