LLM_350M_DENSE

Model Description

LLM_350M_DENSE is a foundational, autoregressive, dense transformer large language model containing approximately 350 million parameters. This model was built, initialized, and trained from scratch using the custom architecture templates provided in the companion firdavsus/LLM_350M_DENSE GitHub repository.

The model design heavily draws optimization inspiration from contemporary architectures (such as Gemma/Llama-style layers), featuring advanced mechanisms like Multi-Query/Grouped-Query Attention, RoPE (Rotary Position Embeddings), and SwiGLU activation functions to maximize training stability and throughput efficiency on standard hardware environments.

Model Features & Specifications

• Parameters: ~350,000,000 (350M)
• Attention Mechanism: Dense Multi-Head/Grouped-Query Attention with FlashAttention-2 compilation bindings.
• Context Length: 2048 / 4096 tokens (depending on exact positional execution adjustments).
• Target Profiles: Multilingual text generation, light reasoning tasks, embedding generations, and low-latency edge deployment.

---

Architectural Details

The core implementation utilizes a decoupled token embedding system paired with the following configurations:

• Hidden Dimension Size: Standard deep hidden layer projection size optimized for intermediate representation bandwidth.
• Activation Layer: SwiGLU (Switching Gated Linear Unit) variants.
• Normalization: RMSNorm applied pre-layer to maintain variance bounding throughout massive token pre-training loops.
• Positional Layout: Rotary Position Embeddings (RoPE) applied to the query and key heads per-layer.

---

Intended Uses & Limitations

Target Applications

• Resource-Constrained Environments: High-throughput inference loops on single customer-grade GPUs or local edge servers.
• Architectural Benchmarking: Serving as a strong, predictable baseline for comparing structural modifications (e.g., Mixture-of-Experts vs. Dense variants).
• Domain Adaptation: Highly suited for continued pre-training on localized languages like Uzbek (uz) or specific code structures.

Limitations

• Parameter Capacity: Given its 350M parameter footprint, the model does not inherently exhibit zero-shot complex mathematical or deep logical multi-step execution paths common in massive 7B+ multi-billion parameter foundation models.
• Tokenizer Alignment: Downstream prompt formats depend directly on the chosen vocabulary generation run (specified inside the training scripts in the GitHub repo).

---

Quickstart Inference

You can initialize and interface with the network components directly via PyTorch using the source scripts in the accompanying repository.

import torch
from model import Transformer, ModelArgs  # Imported from your firdavsus/LLM_350M_DENSE repository
from tokenizer import Tokenizer

# 1. Initialize architectural parameters
args = ModelArgs(
    dim=1024,
    n_layers=24,
    n_heads=16,
    vocab_size=32000,
    max_seq_len=2048
)

# 2. Build model and load checkpoint weights
device = "cuda" if torch.cuda.is_available() else "cpu"
model = Transformer(args).to(device)

checkpoint = torch.load("path_to_model_checkpoint.pt", map_location=device)
model.load_state_dict(checkpoint["model"])
model.eval()

print("Model architecture compiled and weights successfully loaded from disk.")