ForesightLM Core DistilGPT-2
This repository contains the Core ForesightLM checkpoint based on distilgpt2.
ForesightLM studies whether a token-level autoregressive language model can acquire sentence-level foresight through an auxiliary sentence-boundary future semantic objective. The model preserves standard next-token generation while adding a learned projection head for future sentence embedding prediction.
Model components
- Base language model:
distilgpt2 - Core checkpoint: ForesightLM seed 42
- Sentence encoder used during training/evaluation:
sentence-transformers/all-MiniLM-L6-v2 - Future objective: sentence-boundary contrastive future embedding prediction
- Future-loss weight:
lambda_future = 0.08 - Contrastive temperature:
tau = 0.07
Intended use
This checkpoint is intended for research on:
- autoregressive language modeling
- sentence-level semantic planning
- discourse coherence diagnostics
- semantic reranking
- future-representation calibration
Important limitations
This model is a small research prototype. It should not be treated as a production-quality text generator.
Automatic metrics show that semantic reranking is a strong component by itself. Foresight training improves several diagnostics but does not uniformly dominate a reranked baseline. Direct future-head reranking exposes a calibration gap.
Human evaluation protocol files are released in the GitHub repository, but human judgments are still being collected and will be added in a later revision.
Reproducibility
Code, SLURM scripts, evaluation summaries, compute-cost accounting, bootstrap confidence intervals, qualitative examples, and reproducibility manifests are available at:
https://github.com/Ahmet2001/foresightLM
Large generation JSONL files and training data are not included in this model repository.
Citation
If you use this checkpoint, please cite the ForesightLM project repository until a paper DOI/arXiv identifier is available.