Neuvy-TriUnified-MoE-v1

Neuvy-TriUnified-MoE-v1 is a unified Mixture-of-Experts (MoE) Large Language Model engineered by the Neuvy Model Evolution Engine (NMEE).

It synthesizes the architectural strengths of three leading open-weights model paradigms:

1. Meta Llama 3.1 8B: GQA (Grouped-Query Attention) projection topology & reasoning representations.
2. Qwen 2.5 7B: SwiGLU activation geometry, high-density multilingual tokens, and mathematical depth.
3. Mixtral 8x7B & DeepSeek-V3: Block Sparse MoE router with 8 Routed Experts (Top-2 per token) + 1 Shared Expert active on all forward passes.

Architectural Specifications

Quickstart & Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_HF_USERNAME/Neuvy-TriUnified-MoE-v1"

# Load model with custom architecture
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    trust_remote_code=True,
    device_map="auto"
)

# Test forward pass / generation
input_ids = torch.tensor([[1, 15043, 29892, 1128, 505, 338, 263, 1781, 29973]]) # Hello, who are you?
with torch.no_grad():
    outputs = model(input_ids)
    print("Logits shape:", outputs.logits.shape)

Model Lineage & Fusion Architecture

graph TD
    subgraph Parents["Parent Model Contributions"]
        Qwen["Qwen 2.5 7B<br/>(Multilingual & Math)"]
        Llama["Llama 3.1 8B<br/>(Reasoning & GQA)"]
        Mixtral["Mixtral 8x7B<br/>(MoE Routing & General)"]
    end

    subgraph Fusion["Neuvy Model Evolution Engine"]
        SLERP["SLERP Attention Blending<br/>GQA: 12 Query / 4 KV Heads"]
        MoEPool["Expert Pool Synthesis<br/>4 Mixtral + 2 Llama + 2 Qwen"]
        SharedExp["DeepSeek-Style Shared Expert"]
    end

    subgraph Output["Unified Target Model"]
        Target["Neuvy-TriUnified-MoE-v1<br/>8 Routed Experts (Top-2) + 1 Shared Expert"]
    end

    Qwen --> SLERP
    Llama --> SLERP
    Mixtral --> MoEPool
    Llama --> MoEPool
    Qwen --> MoEPool
    SLERP --> Target
    MoEPool --> Target
    SharedExp --> Target

License

Apache 2.0. Developed as part of the CogniOS / Neuvy Operating System ecosystem.