Model Card for Luciole-23B-Base

Table of Contents

• Model Card for Luciole-23B-Base
  • Model Description
  • Uses
    • Direct use
    • Downstream use
    • Out-of-Scope Use
  • Bias, Risks, and Limitations
    • Recommendations
  • Example Code in Python
    • Load the model
    • Sentence completion
  • Loading Intermediate Checkpoints
  • Training Details
    • Training Data
    • Training Procedure
      • Neural Network Architecture
      • Training Hyperparameters
    • Training Convergence and Evaluation
      • Training loss
      • Evaluation
  • Citation
  • Acknowledgements
  • Contact

Model Description

Luciole-23B-Base is a pretrained 23B parameter causal language model developed by LINAGORA and the OpenLLM-France consortium as a part of the OpenLLM France project, funded by BPI France through the France 2030 program. It was trained on the GENCI supercomputer Jean Zay, managed by IDRIS.

Luciole-23B-Base was trained on around 5 trillion tokens of multilingual data, including English (41.9%), French (30.4%), German (3.8%), Spanish (3.5%), Italian (1.9%), Portuguese (1.3%), Dutch (1.0%), Arabic (0.5%), and a small subset of regional languages including regional languages of the French metropolitan area, French variants, and French creoles from around the world (0.4%).

The latter were selected from the FineWeb 2 dataset and include Basque, Breton, Catalan, Corsican, Franco-Provençal, Guadeloupean Creole French, Guianese Creole French, Occitan, Picard, Réunion Creole French, Saint Lucian Creole French, Seselwa Creole French, Tahitian, and Walloon.

Training data also include parallel data from a selection of languages (1.7%), as well as several programming languages (9.2%) and English mathematical data (3.5%).

• License: Apache 2.0
• Training repository: Luciole-Training
• Technical report: coming soon.

Uses

Direct use

Luciole-23B-Base is a foundation language model trained solely to predict the most probable next word in a sequence. It is designed as the first brick in a more complex training pipeline that would include multitask training on diverse instructions or focused fine-tuning on select downstream tasks, as well as possible alignment for human preferences.

Downstream use

Due to its multilingual training, Luciole-23B-Base can be fine-tuned for downstream tasks centered on the generation of multilingual text, with a special focus on French and English.

Out-of-Scope Use

Luciole-23B-Base is not intended to generate text directly for end use cases. It must be fine-tuned first. Its pretraining is optimized for multilingual performance, especially in French and English, and might perform less well on other languages without additional training. While trained on code data, it is not optimized for code generation tasks.

Bias, Risks, and Limitations

Like other foundation models, Luciole-23B-Base is trained on large amounts of web data. Additionally, due to the scarcity of French textual non-web data published under open licenses, much of our French data comes from older works in the public domain that carry biases from other time periods. While we made efforts to reduce toxic and offensive content in the Luciole Training Dataset, Luciole-23B-Base may still generate such content. Filtering of the Luciole Training Dataset is an ongoing project to which we welcome contributions.

Recommendations

To limit the generation of undesirable content, it is advised to fine-tune Luciole-23B-Base through instruction and preference tuning (DPO, RLHF, etc.).

Example Code in Python

Load the model

Load the model (quantized version on GPU if possible, for efficient inference):

import transformers

model_name = "OpenLLM-France/Luciole-23B-Base"

tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
    device_map="auto",
    load_in_4bit=True       # For efficient inference, if quantization is supported by the GPU card
)

Sentence completion

Wrap the model in a text generation pipeline, and specify some generation parameters:

pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)

generation_kwargs = dict(
    num_return_sequences=1,               # Number of variants to generate.
    return_full_text= False,              # Do not include the prompt in the generated text.
    do_sample=True,
    temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
    max_new_tokens=200,                   # Maximum length for the output text (in number of tokens).
)

Try 1-shot question answering:

prompt = """\
Quelle est la capitale de l'Espagne ? Madrid\n\
Quelle est la capitale de la France ?\
"""
completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
    print(prompt + "[…]" + completion['generated_text'])

This will print something like:

Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...

Loading Intermediate Checkpoints

Intermediate checkpoints are released under dedicated revision tags at regular intervals throughout training:

• Every 1,000 steps during the first 5,000 steps
• Then every 5,000 steps up to 30,000 steps
• Then every 10,000 steps beyond that
• In addition, a checkpoint is provided at the end of each training phase

All checkpoints are available at:
https://dl.labs.linagora.com/files/models/OpenLLM-France/Luciole-23B-Base/

They are organized into the following subfolders:

• Phase 1 – Initial pretraining (context length: 4,096)
  From phase1-step0001000 to phase1-step0715787
• Phase 2 – Continued pretraining
  From phase2-step0010000 to phase2-step0358927
• Phase 3 – Annealing phase
  From phase3-annealing-step0010000 to phase3-annealing-step0071526
• Phase 4 – Context extension to 32k tokens
  phase4-context-extension-32k-step0005000 to phase4-context-extension-32k-step0011920
• Phase 5 – Context extension to 131k tokens
  phase5-context-extension-131k-step0005000 to phase5-context-extension-131k-step0011920

The total cumulative number of training steps and training tokens for each checkpoint is specified in the YAML header of each README.md file, and in the config.json file (under the keys "training_steps" and "training_tokens").

Training Details

Training Data

The training dataset used for the pretraining of Luciole-23B-Base is available at OpenLLM-France/Luciole-Training-Dataset. Information on data preprocessing can be found on the data card or in the Luciole-Training repository.

Pretraining consisted of three principal phases of training with a context length of 4,096 tokens. The token breakdowns for the three phases are as follows:

1. Initial pretraining: 3.5 trillion tokens of diverse data
2. Continued pretraining: 1 trillion tokens introducing higher quality data and increasing math and code proportions
3. Annealing phase: 300 billion tokens introducing some instruction-style and reasoning data

Pretraining was followed by two short mid-training phases to extend the context length to 131,072 tokens:

4. First context extension: 50 billion tokens to extend context length from 4,096 to 32,768 tokens
5. Second context extension: 50 billion tokens to extend context length from 32,768 to 131,072 tokens

Training Procedure

Luciole-23B-Base is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).

It was pre-trained on 256 - 512 H100 80GB GPUs (64 - 128 nodes) for about 576,587 GPU hours (1,186 hours) on the Jean Zay supercomputer.

The training code is available at https://github.com/OpenLLM-France/Luciole-Training. Training used version 2.3.1 of NVIDIA's NeMo framework (now transferred to NeMo Automodel or NeMo Megatron-Bridge).

Neural Network Architecture

The architecture of Luciole-23B-Base is a custom adaptation of the Nemotron3-22B recipe. It has 23.2 billion free parameters, with the following hyperparameters:

The "theta" parameter of Rotary Positional Embedding (RoPE) was increased during the context extension phases training process. Its values are indicated in the tables with training hyperparameters below. Note that rotary positional embeddings (RoPE) are applied to the first 50% of the per-head query and key dimensions (rotary_percent = 0.5), with the remaining dimensions left unchanged.

Training Hyperparameters

The details of the intitial pretraining phase are listed below. For each subsequent phase, only the values that differ from the intitial pretraining phase are listed.

1. Initial pretraining

2. Continued Pretraining

3. Annealing

4. Context extension to 32K

5. Context extension to 131K

Training Convergence and Evaluation

Training loss

Information on training loss curves and training stability is available in the training logs, which are released at
metadata/training_logs
├── ConvergenceCurve_phase1.csv -- training logs for phase 1 (initial pretraining)
├── ConvergenceCurve_phase2.csv -- training logs for phase 2 (continued pretraining)
├── ConvergenceCurve_phase3-annealing.csv -- training logs for phase 3 (annealing)
├── ConvergenceCurve_phase4-context-extension-32k.csv -- training logs for phase 4 (context extension to 32k tokens)
└── ConvergenceCurve_phase5-context-extension-131k.csv -- training logs for phase 5 (context extension to 131k tokens)

The following figure shows the training loss curve for the different training phases:

Evaluation

During the training of Luciole-23B-Base, we conducted multiple evaluations to assess performance on standard benchmarks. The primary evaluation languages were French and English, with additional evaluations in German, Spanish, Italian, Portuguese, Dutch, and Arabic. These evaluations were performed on intermediate checkpoints throughout training and on the final checkpoint. We also evaluated the final checkpoint using the RULER benchmark to measure long-context performance.

For comparison, we evaluated the model against the following models, which have between 23B and 2B parameters and were trained on multilingual data, including French:

• Gaperon-1125-24B
• EuroLLM-22B-2512
• Mistral-Small-24B-Base-2501

The main results are summarized in the figures below. Click on a figure to view the complete evaluation results with detailed metrics. The figures illustrate the evolution of evaluation performance across training checkpoints. The x-axis corresponds to the cumulative number of training tokens, while the y-axis reports the average performance across tasks within each category. The RULER figure differs in that it reports average performance across different context lengths.

All figures can be found in the metadata/evaluation folder.

Citation

✍ Paper coming soon!

Acknowledgements

We gratefully acknowledge BPI France for funding the OpenLLM France project under the call "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the France 2030 program.

Training of Luciole-23B-Base was made possible by computing AI and storage resources by GENCI at IDRIS thanks to the grant 2025-AS011016445 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge support from GENCI and IDRIS and from Stephane Requena (GENCI) and Pierre-François Lavallée (IDRIS) in particular.

Luciole-23B-Base was created by members of LINAGORA for the OpenLLM-France project, including in alphabetical order:

Audran Bert
Akshay Chaturvedi
Olivier Gouvert
Julie Hunter
Jean-Pierre Lorré
Jérôme Louradour
Charlotte Noel
Kate Thompson

We thank the support team from IDRIS for technical guidance throughout the project, especially:

Martin Comminges (IDRIS)
Rémi Lacroix (IDRIS)
Myriam Peyrounette (IDRIS)

as well as the support team from NVIDIA, especially:

Meriem Bendris (NVIDIA)
Hayk Shoukourian (NVIDIA)
Oleg Sudakov (NVIDIA)

We are also greatful to the partners of the OpenLLM-France consortium for their valuable input, with particular thanks to (in alphabetical order):

Clément Bénesse (Opsci)
Bertrand Cabot (IDRIS)
Christophe Cerisara (LORIA)
Liam Duignan (CEA)
Olivier Ferret (CEA)
Emile Hazard (OpSci)
Léo Hunout (IDRIS)
Gabriel Lauzzana (LORIA)
Michel-Marie Maudet (LINAGORA)

We would also like to thank members of the Gaperon, Salamandra and Apertus projects for sharing their insights with us. We also acknowledge the numerous open source actors whose resources have guided us throughout the training process, with particular thanks to Nvidia, Hugging Face and Allen AI.

Finally, we thank the entire OpenLLM-France community, whose members have helped in diverse ways.

Contact

contact@openllm-france.fr