LuminoLex-1.7B

LuminoLex-1.7B is an experimental English sparse Mixture-of-Experts (MoE) causal language model developed by LuminoLex AI Lab, pretrained from scratch on 35 billion tokens. It features grouped-query attention (GQA), top-6 routed experts, an always-active shared expert, and tied embeddings.

Highlights

• Parameter Efficiency: 1.7B container capacity with 849.3M effective trained parameters and ~374M active parameters per token.
• Sparse Routing: Routes tokens dynamically across top-6 of 32 active expert banks alongside a dedicated shared expert.
• Consumer Hardware Friendly: Compact footprint designed for low-latency inference on consumer GPUs.

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "LuminoLex/LuminoLex-1.7B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="cuda",
)
model.eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the capital of Japan?"}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        repetition_penalty=1.08,
    )

print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip())

Prompt format:

User: {prompt}
Assistant:

Model Specifications

Architecture & Capacity Note

The checkpoint is packaged in a 1.7B parameter container (1,699,863,296 physical float elements, 20 layers x 64 expert slots) for downstream structural compatibility:

• Trained Layers: 18 active layers contain trained weights (layers 18 and 19 are unpopulated identity padding layers).
• Trained Experts: 32 active routed expert banks are utilized per active layer. The remaining 32 expert slots (32 to 63) are masked via a -9984 router gate bias, preventing them from being selected during inference.
• Active Parameters: In active computation, each token engages the embeddings, shared backbone, and top-6 active experts across the 18 trained layers, executing exactly 373,920,384 active parameters per token.

Local Gradio UI

Run the local chat interface:

pip install -r requirements.txt
python app.py

Limitations

LuminoLex-1.7B is a compact experimental model. Outputs may be inaccurate, repetitive, or inconsistent on complex multi-step reasoning, arithmetic, strict formatting, and code generation. Intended for research and evaluation purposes.