Project links: SLM-Tavenel overview · published model catalog · research and reproducibility archive. This card describes this checkpoint; the repository overview links the related models and broader experiment record.
Experimental English/French base model (1.18M parameters), trained from scratch on a CC0 subset of
French-Science-Commons, released with a preregistered ten-seed study of its head-wise attention gate.
Not instruction tuned; not a chat assistant. Project self-assessment: the package meets the core
requirements of the Open Source AI Definition 1.0 under MIT terms for weights, code and Data Information
(training_record/OSAID-1.0-SELF-ASSESSMENT.md); this is not OSI certification. Source-license metadata
and privacy limitations are disclosed below.
| At a glance | |
|---|---|
| Parameters | 1,181,920 (ungated control: 1,180,768) |
| Architecture | Decoder-only transformer, 4 layers, width 96, 3 query / 1 key-value heads, SwiGLU 256, head-wise attention gate |
| Tokenizer | Project byte-level BPE, 8,192 tokens (tokenizers 0.22.2) |
| Context window | 1,024 tokens |
| Training precision | bfloat16 mixed precision, deterministic algorithms |
| Published weights | model.safetensors, float32 (SHA-256 b2ef1606…); all 20 study runs under training_record/replicates/; bfloat16 copy staged in variants/bf16/ |
| Training data | 1,229 CC0-labelled French-Science-Commons documents (748 EN, 481 FR); 15.98M training-token positions (~one pass) |
| Optimizer / seed | AdamW; release seed 421101, preselected in the protocol |
| Compute | One RTX 4060 Laptop GPU (8 GB); ~0.06 GPU-hours for the released run (20 runs in the study); energy not measured |
| License | MIT (weights, code, Data Information) |
| Hub revision described | 33ddccd588a693efeaceae9ae239fa0fbd0c94b6 (weights identical at later revisions) |
Main result · Benchmarks · Local inference · Use · Architecture · Training · Traceability and explainability · Intended use and safety · Data, licenses and privacy · Limitations · License and citation · Model family
Across ten paired seeds, the gated model reduced held-out bits per byte by 0.0275 relative to a matched ungated model (hierarchical bootstrap 95% interval [0.0168, 0.0395]; predeclared practical threshold 0.01 BPB). The primary metric is the equal-language macro-average of English and French BPB on a document-disjoint test set; the effect is ungated minus gated BPB, so a positive value favors the gate; the interval resamples paired seeds and test documents within language. The finding applies to this corpus, tokenizer and one-pass budget only.
Per-seed effects and the pooled interval. Per-seed values, document-hash-level scores and the locked
analysis: training_record/sealed-test-bpb-results.json.
Protocol. Zero-shot multiple choice scored by log-likelihood with the SLM-Tavenel evaluator on pinned
items (lm-evaluation-harness prompt templates for ARC, HellaSwag and Belebele; cloze scoring for
WinoGrande). Post-hoc and descriptive: no family-wise correction. Paired item-bootstrap differences
against the 10M and four public references: training_record/benchmark-comparisons.json.
| Model | Parameters | ARC-Easy | ARC-Challenge | HellaSwag | Belebele EN | Belebele FR | WinoGrande | Six-task mean |
|---|---|---|---|---|---|---|---|---|
| SLM-Tavenel 1M | 1.18M | 0.280 | 0.196 | 0.259 | 0.264 | 0.266 | 0.493 | 0.293 |
| SLM-Tavenel 10M | 9.98M | 0.306 | 0.184 | 0.275 | 0.296 | 0.319 | 0.511 | 0.315 |
| SLM-Tavenel 25M | 25.7M | 0.319 | 0.180 | 0.273 | 0.283 | 0.309 | 0.519 | 0.314 |
| SLM-Tavenel 25M HPLT 10% | 25.7M | 0.319 | 0.187 | 0.274 | 0.279 | 0.308 | 0.517 | 0.314 |
| SLM-Tavenel 124M | 124M | 0.484 | 0.206 | 0.277 | 0.316 | 0.311 | 0.519 | 0.352 |
| TinyStories tiny1m | 0.94M | 0.256 | 0.189 | 0.260 | 0.274 | 0.290 | 0.518 | 0.298 |
| Pythia-14M | 14.1M | 0.330 | 0.161 | 0.262 | 0.256 | 0.309 | 0.520 | 0.306 |
| Pythia-70M | 70M | 0.317 | 0.194 | 0.268 | 0.282 | 0.306 | 0.487 | 0.309 |
| GPT-2 Small | 124M | 0.447 | 0.185 | 0.292 | 0.310 | 0.318 | 0.517 | 0.345 |
| Pythia-160M | 160M | 0.386 | 0.187 | 0.285 | 0.288 | 0.309 | 0.493 | 0.324 |
| SmolLM2-135M | 135M | 0.643 | 0.278 | 0.356 | 0.358 | 0.314 | 0.530 | 0.413 |
| Items | 2,376 | 1,172 | 10,042 | 900 | 900 | 1,267 | ||
| 95% binomial half-width (typical) | ±0.019 | ±0.022 | ±0.009 | ±0.030 | ±0.030 | ±0.028 | ||
| Chance | 0.25 | 0.25 | 0.25 | 0.25 | 0.25 | 0.50 |
Raw zero-shot accuracy, all rows scored by the SLM-Tavenel log-likelihood evaluator on the same pinned
items (ARC allenai/ai2_arc@210d026f, HellaSwag Rowan/hellaswag@218ec52e validation, Belebele
facebook/belebele@7899cdfa, WinoGrande XL allenai/winogrande@546d5aa0 validation). Half-widths are
item-sampling intervals for one checkpoint; they do not include training-seed variation, and paired
intervals (in each card) are the right tool for comparing two models on the same items. The models
differ in data, tokenizer, training budget and possible benchmark exposure: this table describes
checkpoints, it is not a ranking of recipes.
Measured on NVIDIA GeForce RTX 4060 Laptop GPU (8 GB) and an Intel Core i7-12650H (10 cores, 68 GB RAM), Windows 11, Python 3.11.9, PyTorch 2.10.0+cu130, Transformers 4.57.6. Protocol: batch size 1, 128-token prompt, greedy decoding of exactly 64 new tokens, one warm-up run, median of timed runs (scripts/benchmark_card_inference.py, scripts/verify_kv_cache_release.py; raw results in inference-benchmark.json and kv-cache-verification.json).
Generation speed, with and without the KV cache (generated tokens per second):
| Device | Without cache | With KV cache | Speed-up | Same tokens with and without cache |
|---|---|---|---|---|
| CPU, float32 | 155.0 | 286.1 | ×1.85 | yes |
| GPU, float32 | 131.6 | 136.4 | ×1.04 | yes |
Cached generation gives the same logits as a full-prefix, no-cache reference (maximum absolute difference 0) and the same greedy tokens with and without the cache. It speeds up CPU generation; on this GPU, small models are limited by per-token kernel-launch overhead rather than arithmetic, so the cache brings no gain there.
Latency and memory (code without cache, as first published):
| Device | Weights dtype | Prefill, 128 tokens (≈ time to first token) | 64 new tokens, total | Peak process RAM | Peak allocated VRAM |
|---|---|---|---|---|---|
| CPU | float32 | 4.8 ms | 0.37 s | 0.83 GB | — |
| GPU | float32 | 5.5 ms | 0.46 s | 1.35 GB | 20 MB |
| GPU | bfloat16 | 6.4 ms | 0.58 s | 1.48 GB | 15 MB |
Peak process RAM includes the Python and PyTorch runtime (about 0.8 GB on CPU, 1.35 GB once CUDA is initialized), so it overstates the model's own footprint.
Reduced-precision variants (scripts/quantize_and_measure.py, quantization-measurements.json). Quality on one fixed text sample (first 100 English + 100 French documents of the SLM-Tavenel corpus-v2 held-out test split; only the change against float32 is meaningful):
| Variant | Device | Weights size | BPB on the sample | Change vs float32 | Next-token top-1 agreement with float32 | Tokens/s (KV cache) |
|---|---|---|---|---|---|---|
| float32 | CPU | 4.7 MB | 2.4715 | +0.0000 | 100.0% | 250.4 |
| bfloat16 | GPU | 2.4 MB | 2.4716 | +0.0001 | 98.0% | 113.3 |
| int8-dynamic | CPU | 4.3 MB (in memory) | 2.4753 | +0.0038 | 89.9% | 129.7 |
bfloat16: weights cast to bfloat16 (variants/bf16/, half the file size). int8-dynamic: PyTorch dynamic quantization of every linear layer, applied at load time on CPU (torch.ao.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)); it is not stored as a file. No 4-bit, GGUF or bitsandbytes variant has been produced or measured.
Single machine, single session: treat the timings as indicative.
pip install -r requirements.txt # versions recorded with this package
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "athenista/slm-tavenel-1m"
revision = "33ddccd588a693efeaceae9ae239fa0fbd0c94b6"
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, revision=revision, trust_remote_code=True)
inputs = tokenizer("La recherche scientifique", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=48, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
trust_remote_code=True is required (custom architecture; hf_model.py, hf_config.py; read it
before running). The revision pinned above already contains the KV-cache implementation used for the measurements in Local inference. The cache was checked against uncached generation for this checkpoint; see the recorded logits and token-equivalence results.
Four pre-RMSNorm decoder blocks with grouped-query causal attention, QK normalization, RoPE, a head-wise sigmoid gate on attention outputs and a SwiGLU feed-forward sublayer, with residual connections.
| Setting | Value |
|---|---|
| Design | Ten paired seeds: gated vs ungated, identical initial tensors except gate parameters, identical seed-indexed batches |
| Optimizer | AdamW, β = (0.9, 0.95), weight decay 0.1, gradient clipping 1.0 |
| Schedule | Warmup–stable–decay: 48 warmup steps, peak 3e-3, final 20% decay to 3e-4 |
| Batch | 16 sequences × 1,024 tokens × 2 accumulation steps = 32,768 token positions per step |
| Steps / tokens | 488 steps, 15,990,784 token presentations |
| Precision | bfloat16 autocast, deterministic algorithms |
| Code revision | 02396acd15ef9bcdc453bcd53aade40819e2f69f, clean worktree |
| Hardware | 1 × NVIDIA GeForce RTX 4060 Laptop GPU (8 GB), peak allocated 1.9 GB |
| Software | Python 3.11.9, PyTorch 2.10.0+cu130, CUDA 13.0, tokenizers 0.22.2 |
| Compute | ~0.06 GPU-hours for the released run, summed from the logged throughput (the 20-run study is about 20 times that) |
| Energy / emissions | Not measured for this run (no power log was recorded); not estimated here. Training runs from 2026-11-06 on log GPU board energy through NVML (src/slm/energy.py) |
Traceability. The bundle includes a record-level manifest (source IDs and URIs, titles, declared
authors, publication dates, DOI where supplied, declared license labels, language, split, pre/post-
processing text hashes), the source revision, HAL snapshot, filtering decisions, pipeline configuration,
split and tokenizer manifests, privacy-screen report, processing scripts and provenance.json. The data
pipeline ran in a dirty worktree; it was replayed from the recorded inputs into a scratch directory and
the retained-document file, decisions and statistics matched the original hashes exactly (replay
configuration, manifest and comparison report included). The training runs used a clean commit.
Explainability (observability, not causal explanation). Each run's
training_record/replicates/<seed>/<arm>/observability.jsonl records, on a fixed probe batch,
attention-sink mass, attention entropy, residual statistics, gate values, per-layer weight norms and
update ratios. The released gated run's last record:
The gated/ungated pairs make a controlled comparison of these diagnostics possible (same data, initialization and steps). They are measurements, not explanations of particular outputs, and were not validated by interventions.
scripts/safety_eval_basic.py, results in safety-eval-basic.json), not a safety certification. Toxicity: 500 prompts sampled from the "challenging" subset of RealToxicityPrompts (English; Gehman et al., 2020), one sampled continuation of 32 tokens each (top-p 0.9), scored by unitary/toxic-bert. Personal data: 20 English/French prompts inviting contact details × 10 samples, scanned with the project's PII detector (email, phone, IPv4, IBAN). The prompts themselves score 0.77 mean toxicity.| Model | Mean toxicity [95% CI] | Continuations with toxicity > 0.5 [95% CI] | PII-shaped outputs | Redaction placeholders (<EMAIL>…) |
|---|---|---|---|---|
| slm-tavenel-1m | 0.018 [0.010, 0.026] | 0.8% [0.2%, 1.6%] | 0.0% | 2.0% |
| GPT-2 Small (same harness, reference) | 0.417 [0.378, 0.455] | 42.2% [38.0%, 46.6%] | 21.0% | 0.0% |
Lower toxicity than GPT-2 is not evidence of safety: small or domain-specific models often produce less fluent continuations, which classifiers score as less toxic. The PII probe detects strings shaped like personal data; it does not check whether they belong to real people. Not covered: French toxicity (no French-capable classifier with a compatible license was found), bias, memorization of training text, jailbreaks. No instruction, preference or safety tuning was applied.
Source: the french_open_science subset of
PleIAs/French-Science-Commons, revision
bfa114c09d7c28dcd0a8a47fe445c64b4a3c6156. Of 3,483 English/French records labelled CC0, a paginated HAL
audit (2026-10-22) found 3,450 whose licence_s main-file license and all reported fileLicenses_s
document-file licenses were CC0 across returned versions; 33 were excluded for missing, conflicting or
non-CC0 HAL records. The filters retained 1,229 documents (748 English, 481 French). Training split
15,976,924 tokens; development 931,435; document-held-out test 2,032,436 (69 English and 47 French
documents, text not included). The audit establishes metadata agreement, not depositor authority or
publisher permission (HAL licence API announcement;
audit note).
The pipeline redacted 2,579 email-shaped strings, 158 phone-shaped strings and three IP-pattern matches, and excluded five documents with residual email-shaped strings and two with replacement characters. The retained text had zero matches for the configured residual scan patterns; a deterministic screen reviewed 10 short excerpts per language. These are limited checks, not proof that the corpus contains no personal or sensitive information.
Weights, inference implementation, training code and Data Information: MIT. The MIT notices do not
relicense source articles or guarantee the absence of third-party claims. Cite the SLM-Tavenel
repository, Hub revision 33ddccd588a693efeaceae9ae239fa0fbd0c94b6, and training_record/results-report.md,
training_record/protocol.md and training_record/replication-driver-record.json. No paper or DOI is
assigned.
| Model | Parameters | Data | Relation |
|---|---|---|---|
| slm-tavenel-1m | 1.18M | French-Science-Commons (CC0 subset) | ten-seed gate study |
| slm-tavenel-10m | 9.98M | French-Science-Commons | trained from scratch |
| slm-tavenel-25m | 25.7M | French-Science-Commons | grown from 10M |
| slm-tavenel-25m-hplt-10pct | 25.7M | + 10% HPLT | continuation of 25M |
| slm-tavenel-124m | 124M | FineWeb-Edu + FineWeb2-HQ | trained from scratch, new tokenizer |
training_record/replicates/seed-421101/gated.safetensors 5 MBtraining_record/replicates/seed-421102/gated.safetensors 5 MBtraining_record/replicates/seed-421103/gated.safetensors 5 MBtraining_record/replicates/seed-421104/gated.safetensors 5 MBtraining_record/replicates/seed-421105/gated.safetensors 5 MBtraining_record/replicates/seed-421106/gated.safetensors 5 MBtraining_record/replicates/seed-421107/gated.safetensors 5 MBtraining_record/replicates/seed-421108/gated.safetensors 5 MBtraining_record/replicates/seed-421109/gated.safetensors 5 MBtraining_record/replicates/seed-421110/gated.safetensors 5 MBmodel.safetensors 5 MBtraining_record/replicates/seed-421101/ungated.safetensors 5 MBtraining_record/replicates/seed-421102/ungated.safetensors 5 MBtraining_record/replicates/seed-421103/ungated.safetensors 5 MBtraining_record/replicates/seed-421104/ungated.safetensors 5 MBtraining_record/replicates/seed-421105/ungated.safetensors 5 MBtraining_record/replicates/seed-421106/ungated.safetensors 5 MBtraining_record/replicates/seed-421107/ungated.safetensors 5 MBtraining_record/replicates/seed-421108/ungated.safetensors 5 MBtraining_record/replicates/seed-421109/ungated.safetensors 5 MBtraining_record/replicates/seed-421110/ungated.safetensors 5 MB