openeurollm/kda-sigmoid-hybrid-1.3B-100B

Paper: https://arxiv.org/abs/2609.24797

Code: https://github.com/OpenEuroLLM/ComplexKDA

Authors: Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein

A KDA baseline (bounded sigmoid gate) hybrid language model (1.36B parameters) -- linear layers with full attention every 4th layer -- from the ComplexKDA release.

ComplexKDA is Kimi Delta Attention with a signed decay gate: the per-channel decay alpha is allowed to take either sign, alpha in [-1, 1], instead of being confined to (0, 1]. That is the one-dimensional real case of a complex eigenvalue, so a channel can oscillate rather than only forget. The magnitude is carried in log space exactly as KDA carries it; the +-1 part is carried as a running product pushed onto the queries and keys, so the recurrence the kernels run is still the unsigned one.

This checkpoint's decay gate is unsigned sigmoid (the KDA baseline: alpha in (0, 1]).

Architecture

• 24 layers, hidden size 2048, MLP 5312 (SwiGLU)
• 16 heads of dimension 128, short convolution of width 4
• vocabulary 32000, trained at context 4096
• embeddings untied
• attention at layers [3, 7, 11, 15, 19, 23] (gated, NoPE), linear everywhere else

Tokenizer, and how to start a prompt

The bundled tokenizer is configured the way the training corpus was encoded: no BOS is prepended, documents were terminated with the EOS token, and model_max_length is this model's trained context. The upstream tokenizer repository's own defaults differ on both points, so encode through the tokenizer shipped here rather than re-fetching it by name.

To condition on the start of a document, prefix the EOS token -- that is what precedes every document's first token in training, and a BOS was never seen at any position. Leave a continuation bare: mid-document the prefix is a false signal and costs accuracy. bos_token is remapped to </s> here, so a caller that asks for "the BOS" gets the separator, while add_bos_token stays False and the default remains a bare prompt.

Usage

The bundled modeling_complex_kda.py is standalone: torch and transformers are all it needs.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("openeurollm/kda-sigmoid-hybrid-1.3B-100B")
model = AutoModelForCausalLM.from_pretrained(
    "openeurollm/kda-sigmoid-hybrid-1.3B-100B", trust_remote_code=True, dtype="bfloat16")

For the Triton kernels these models were trained with -- much faster, and the exact code path of the training runs -- install the fork:

pip install git+https://github.com/OpenEuroLLM/ComplexKDA

It is picked up automatically when importable. COMPLEX_KDA_BACKEND=torch forces the portable path; =kernel makes a missing fork an error instead of a silent fallback.

Provenance

Converted from the training checkpoint with lm_scaling/hf_release/convert_to_hub.py. The conversion is metadata only -- the weight file is the exporter's own, byte for byte -- and the bundled implementation is checked against the reference implementation the runs used.