Luna 3 Model Card

Model Information

• Model creator: AobanZ

Description

Luna 3 is our next generation text model designed for heavy usage and lengthy conversations. These models have added Llama.cpp support as well as faster response time. Luna has an 8k context window, which supports long chats and roleplays. Luna 3 retains the Lightning architecture, with the only changes being upgraded RMSNorm and RoPE. Luna 3 generates at a relatively fast speed of 130 tokens per second.

• Developed by: AobanZ
• Model type: Transformer
• Language(s) (NLP): English
• License: MIT

[!IMPORTANT] If you try installing llama.cpp traditionally, the program will exit with unknown model architecture: 'lightning'. A patch of llama.cpp has been released called Lightning.cpp that supports the Luna architecture. Lightning.cpp

Model Sources

• Repository: https://huggingface.co/Aobangaming/luna-2-70m

Highlights

• Faster Response Time With Llama.cpp, Luna generates tokens faster and more efficiently.
• Increased Intelligence Luna was sized up from 100m to 230m, which leads to around 100M transformer parameters.

Usage

Luna can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities. Direct use is primarily intended for casual conversation, research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.

Recommendations

We recommend users of Luna to fine-tune the model on new text, and add necessary guardrails and precautions to prevent misuse.

Starter

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

REPO = "Aobangaming/Luna-3-230m" 

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(
    REPO, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda").eval()

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Hi"}],
    tokenize=False, add_generation_prompt=True,
)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")

with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=512)

print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Training Details

Training Data

Luna was trained on a subset of the OASST dataset.

Training Procedure

Luna 3 was trained on an RTX 3050 GPU, using FlashAttention/SDPA and MHA. The model was trained on OpenAssistant. Loss ended at around 1.9. Luna 3 is a fork of Luna 2, simply with upgraded sequence length and support.

Training Hyperparameters

Hardware & Infrastructure

• Hardware Type: RTX 3050 6GB
• Hours used: 5
• Cloud Provider: AobanLabs
• Compute Region: Asia

Technical Specifications

Model Architecture and Objective

Luna uses a 12-layer causal Transformer with 768-dimensional hidden states and 12 attention heads, with RMSNorm and RoPE. The architecture uses Multi-Head attention and Scaled Dot product attention. Luna 3 is designed to be a fast, efficient alternative to larger models like Llama and Qwen.