Model Card for Luciole-8B-Base
Table of Contents
- Model Card for Luciole-8B-Base
Model Description
Luciole-8B-Base is a pretrained 8B parameter causal language model with a hybrid Mamba-transformer architecture. It was developed by LINAGORA and the OpenLLM-France consortium as a part of the OpenLLM France project, funded by BPI France through the France 2030 program. It was trained on the GENCI supercomputer Jean Zay, managed by IDRIS.
Luciole-8B-Base was trained on around 5 trillion tokens of multilingual data, including English (41.9%), French (30.4%), German (3.8%), Spanish (3.5%), Italian (1.9%), Portuguese (1.3%), Dutch (1.0%), Arabic (0.5%), and a small subset of regional languages including regional languages of the French metropolitan area, French variants, and French creoles from around the world (0.4%).
The latter were selected from the FineWeb 2 dataset and include Basque, Breton, Catalan, Corsican, Franco-Provençal, Guadeloupean Creole French, Guianese Creole French, Occitan, Picard, Réunion Creole French, Saint Lucian Creole French, Seselwa Creole French, Tahitian, and Walloon.
Training data also include parallel data from a selection of languages (1.7%), as well as several programming languages (9.2%) and English mathematical data (3.5%).
- License: Apache 2.0
- Training repository: Luciole-Training
- Technical report: coming soon.
Uses
Direct use
Luciole-8B-Base is a foundation language model trained solely to predict the most probable next word in a sequence. It is designed as the first brick in a more complex training pipeline that would include multitask training on diverse instructions or focused fine-tuning on select downstream tasks, as well as possible alignment for human preferences.
Downstream use
Due to its multilingual training, Luciole-8B-Base can be fine-tuned for downstream tasks centered on the generation of multilingual text, with a special focus on French and English.
Out-of-Scope Use
Luciole-8B-Base is not intended to generate text directly for end use cases. It must be fine-tuned first. Its pretraining is optimized for multilingual performance, especially in French and English, and might perform less well on other languages without additional training. While trained on code data, it is not optimized for code generation tasks.
Bias, Risks, and Limitations
Like other foundation models, Luciole-8B-Base is trained on large amounts of web data. Additionally, due to the scarcity of French textual non-web data published under open licenses, much of our French data comes from older works in the public domain that carry biases from other time periods. While we made efforts to reduce toxic and offensive content in the Luciole Training Dataset, Luciole-8B-Base may still generate such content. Filtering of the Luciole Training Dataset is an ongoing project to which we welcome contributions.
Recommendations
To limit the generation of undesirable content, it is advised to fine-tune Luciole-8B-Base through instruction and preference tuning (DPO, RLHF, etc.).
Example Code in Python
Load the model
Load the model (quantized version on GPU if possible, for efficient inference):
import transformers
model_name = "OpenLLM-France/Luciole-8B-Base"
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
device_map="auto",
load_in_4bit=True # For efficient inference, if quantization is supported by the GPU card
)
Sentence completion
Wrap the model in a text generation pipeline, and specify some generation parameters:
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
generation_kwargs = dict(
num_return_sequences=1, # Number of variants to generate.
return_full_text= False, # Do not include the prompt in the generated text.
do_sample=True,
temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
max_new_tokens=200, # Maximum length for the output text (in number of tokens).
)
Try 1-shot question answering:
prompt = """\
Quelle est la capitale de l'Espagne ? Madrid\n\
Quelle est la capitale de la France ?\
"""
completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
print(prompt + "[…]" + completion['generated_text'])
This will print something like:
Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...
Loading Intermediate Checkpoints
Intermediate checkpoints are released under dedicated revision tags at regular intervals throughout training:
- Every 1,000 steps during the first 5,000 steps
- Then every 5,000 steps up to 30,000 steps
- Then every 10,000 steps beyond that
- In addition, a checkpoint is provided at the end of each training phase
All checkpoints are available at:
https://dl.labs.linagora.com/files/models/OpenLLM-France/Luciole-8B-Base/
They are organized into the following subfolders:
- Phase 1 – Initial pretraining (context length: 4,096)
From phase1-step0001000 to phase1-step0715787 - Phase 2 – Continued pretraining
From phase2-step0010000 to phase2-step0358930 - Phase 3 – Annealing phase
From phase3-annealing-step0010000 to phase3-annealing-step0118238 - Phase 4 – Context extension to 131k tokens
phase4-context-extension-131k-step0003000 to phase4-context-extension-131k-step0023841
The total cumulative number of training steps and training tokens for each checkpoint is specified in
the YAML header of each README.md file, and in
the config.json file (under the keys "training_steps" and "training_tokens").
Training Details
Training Data
The training dataset used for the pretraining of Luciole-8B-Base is available at OpenLLM-France/Luciole-Training-Dataset. Information on data preprocessing can be found on the data card or in the Luciole-Training repository.
Pretraining consisted of three principal phases of training with a context length of 4,096 tokens. The token breakdowns for the three phases are as follows:
- Initial pretraining: 3.5 trillion tokens of diverse data
- Continued pretraining: 1 trillion tokens introducing higher quality data and increasing math and code proportions
- Annealing phase: 0.5 trillion tokens introducing some instruction-style and reasoning data
Pretraining was followed by one short mid-training phases to extend the context length to 131,072 tokens:
- Context extension: 100 billion tokens to extend context length from 4,096 to 131,072 tokens
Training Procedure
Luciole-8B-Base is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).
It was pre-trained on 128 - 256 H100 80GB GPUs (32 - 64 nodes) for about 237,036 GPU hours (870 hours) on the Jean Zay supercomputer.
The training code is available at https://github.com/OpenLLM-France/Luciole-Training. Training used version 2.3.1 of NVIDIA's NeMo framework (now transferred to NeMo Automodel or NeMo Megatron-Bridge).
Neural Network Architecture
The architecture of Luciole-8B-Base is a custom adaptation of the NemotronH-8B recipe. It has exactly 8.08 billion free parameters, with the following hyperparameters:
| Hyperparameter | Value |
|---|---|
| Vocabulary size (# tokens) | 128,000 |
| # blocks | 52 |
| # attention heads | 32 |
| # key-value heads | 8 |
| Hidden size | 4096 |
| Intermediate size | 21504 |
| SSM State size | 128 |
| Mamba # heads | 128 |
| MLP Activation | relu2 |
| Mamba Activation | silu |
Training Hyperparameters
The details of the intitial pretraining phase are listed below. For each subsequent phase, only the values that differ from the intitial pretraining phase are listed.
1. Initial pretraining
| Hyperparameter | Value |
|---|---|
| Total # samples | 732,965,888 (3T tokens) |
| Total # steps | 715,787 |
| Context length | 4,096 |
| Batch size | 1,024 |
| Learning rate schedule | Warmup (2M samples) + Constant |
| Learning rate | 3e-4 |
| Weight decay | 0.1 |
| Dropout | _ |
| Gradient clipping | 1 |
| Initializer range | 0.009 |
| Optimizer | AdamW (β₁=0.9, β₂=0.95, ε=1e-5) |
| Precision | bfloat16 |
| Tensor Parallelism (with 256 GPUs) | 2 |
| Pipeline Parallelism (with 256 GPUs) | 1 |
| Data Parallelism (with 256 GPUs) | 128 |
2. Continued Pretraining
| Hyperparameter | Value |
|---|---|
| Total # samples | 367,545,344 (1.5T tokens) |
| Total # steps | 358,931 |
| Learning rate schedule | Cosine annealing |
| Maximum Learning rate | 3e-4 |
| Final Learning rate | 6.87e-5 |
3. Annealing
| Hyperparameter | Value |
|---|---|
| Total # samples | 121,075,712 (0.5T tokens) |
| Total # steps | 118,238 |
| Learning rate schedule | Linear annealing |
| Maximum Learning rate | 6.87e-5 |
| Final Learning rate | 0 |
4. Context extension to 131K
| Hyperparameter | Value |
|---|---|
| Total # samples | 762,912 (100B tokens) |
| Total # steps | 23,841 |
| Context length | 131,072 |
| Batch size | 32 |
| Context Parallelism (with 128 GPUs) | 8 |
| Tensor Parallelism (with 128 GPUs) | 2 |
| Data Parallelism (with 128 GPUs) | 8 |
Training Convergence and Evaluation
Training loss
Information on training loss curves and training stability is available in the training logs, which are released at
metadata/training_logs
├── ConvergenceCurve_phase1.csv -- training logs for phase 1 (initial pretraining)
├── ConvergenceCurve_phase2.csv -- training logs for phase 2 (continued pretraining)
├── ConvergenceCurve_phase3-annealing.csv -- training logs for phase 3 (annealing)
└── ConvergenceCurve_phase4-context-extension-131k.csv -- training logs for phase 4 (context extension to 131k tokens)
The following figure shows the training loss curve for the different training phases:
Evaluation
During the training of Luciole-8B-Base, we conducted multiple evaluations to assess performance on standard benchmarks. The primary evaluation languages were French and English, with additional evaluations in German, Spanish, Italian, Portuguese, Dutch, and Arabic. These evaluations were performed on intermediate checkpoints throughout training and on the final checkpoint. We also evaluated the final checkpoint using the RULER benchmark to measure long-context performance.
For comparison, we evaluated the model against the following models, which have between 8B and 2B parameters and were trained on multilingual data, including French:
- Gaperon-1125-8B
- EuroLLM-9B
- salamandra-7b
- Llama-3.1-8B
- Apertus-8B-2509
- Qwen3-8B-Base
- Ministral-3-8B-Base-2512
The main results are summarized in the figures below. Click on a figure to view the complete evaluation results with detailed metrics. The figures illustrate the evolution of evaluation performance across training checkpoints. The x-axis corresponds to the cumulative number of training tokens, while the y-axis reports the average performance across tasks within each category. The RULER figure differs in that it reports average performance across different context lengths.
All figures can be found in the metadata/evaluation folder.
Citation
✍ Paper coming soon!
Acknowledgements
We gratefully acknowledge BPI France for funding the OpenLLM France project under the call "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the France 2030 program.
Training of Luciole-8B-Base was made possible by computing AI and storage resources by GENCI at IDRIS thanks to the grant 2025-AS011016445 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge support from GENCI and IDRIS and from Stephane Requena (GENCI) and Pierre-François Lavallée (IDRIS) in particular.
Luciole-8B-Base was created by members of LINAGORA for the OpenLLM-France project, including in alphabetical order:
Audran Bert
Akshay Chaturvedi
Olivier Gouvert
Julie Hunter
Jean-Pierre Lorré
Jérôme Louradour
Charlotte Noel
Kate Thompson
We thank the support team from IDRIS for technical guidance throughout the project, especially:
Martin Comminges (IDRIS)
Rémi Lacroix (IDRIS)
Myriam Peyrounette (IDRIS)
as well as the support team from NVIDIA, especially:
Meriem Bendris (NVIDIA)
Hayk Shoukourian (NVIDIA)
Oleg Sudakov (NVIDIA)
We are also greatful to the partners of the OpenLLM-France consortium for their valuable input, with particular thanks to (in alphabetical order):
Clément Bénesse (Opsci)
Bertrand Cabot (IDRIS)
Christophe Cerisara (LORIA)
Liam Duignan (CEA)
Olivier Ferret (CEA)
Emile Hazard (OpSci)
Léo Hunout (IDRIS)
Gabriel Lauzzana (LORIA)
Michel-Marie Maudet (LINAGORA)
We would also like to thank members of the Gaperon, Salamandra and Apertus projects for sharing their insights with us. We also acknowledge the numerous open source actors whose resources have guided us throughout the training process, with particular thanks to Nvidia, Hugging Face and Allen AI.
Finally, we thank the entire OpenLLM-France community, whose members have helped in diverse ways.