LFM3-Omni Dual-Head Architecture

Unified System 1 (Fast Decision) & System 2 (Multimodal Generation) on a Shared Backbone

This repository introduces the first unified Dual-Head architecture combining:

1. System 1 (Decision Model): Fast, non-autoregressive decision-making based on LiquidAI/d1-Omni-2B (zero output tokens, single-pass classification/routing).
2. System 2 (Generative Model): Autoregressive text and speech generation based on LiquidAI/LFM3-Omni-2B.

---

💡 Why Dual-Head?

Previously, developers needing both instant event triage/routing and conversational abilities had to load two separate 2.2B models into GPU memory, consuming 8–10 GB of VRAM.

Because both models share the same Tri-hybrid LFM3-Omni trunk (gated short convolutions, sliding-window attention, and global attention), this architecture unifies them into a single object:

• ⚡ ~50% VRAM Savings: Uses only ~4.5 GB VRAM (or ~1.8 GB in 4-bit quantization).
• 🧠 Zero Weight Duplication: Shares projection tensors in memory directly using memory views.
• 🎯 Dual Dispatch: Call .system_one(...) for sub-50ms decisions, and .generate(...) for full chat and multimodal responses.

---

🏛 Architecture Diagram

                        [ Input: Text / Image / Audio ]
                                       │
                                       ▼
             ┌──────────────────────────────────────────────────┐
             │       Shared LFM3-Omni Backbone (~2.1B)         │
             │   (Tri-hybrid: Gated Conv + Sliding Attention)   │
             └────────────────────────┬─────────────────────────┘
                                      │
                 ┌────────────────────┴────────────────────┐
                 ▼                                         ▼
   ┌───────────────────────────┐             ┌───────────────────────────┐
   │       Decision Head       │             │       Language Head       │
   │      (from d1-Omni)       │             │     (from LFM3-Omni)      │
   │  Call via .system_one()   │             │   Call via .generate()    │
   │  Output: Typed Decisions  │             │   Output: Text & Speech   │
   └───────────────────────────┘             └───────────────────────────┘

---

🚀 Quickstart & Usage

1. Installation

pip install -U "transformers>=4.40" torch pillow numpy huggingface_hub

2. Loading the Model

import torch
from transformers import AutoModel

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    "JonusNattapong/lfm3-omni-dualhead",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
    device_map="auto" if device == "cuda" else None
)

3. System 1: Fast Calibrated Decision (Single Forward Pass)

# Triage customer intent & urgency in sub-50ms (0 output tokens)
state_text = "Urgent: payment webhook failed for order #1234"

questions = {
    "intent": {
        "type": "choice",
        "choices": ["payment_issue", "account_inquiry", "general_support"]
    },
    "is_urgent": {
        "type": "boolean"
    }
}

result = model.system_one(state=state_text, questions=questions)
print(result)
# Output: {'intent': 'payment_issue', 'is_urgent': True}

4. Multimodal Image Decision (224x224)

from PIL import Image

image = Image.open("sample.jpg").resize((224, 224))
questions = {
    "has_receipt": {"type": "boolean"},
    "quality_score": {"type": "number", "range": [1, 10], "step": 1}
}

image_decision = model.system_one(media={"image": image}, questions=questions)
print(image_decision)

5. Multimodal Audio Event Routing (16kHz Audio)

import numpy as np

# Audio clip (16 kHz mono)
rate = 16000
audio_clip = np.zeros(2 * rate, dtype=np.float32)

questions_audio = {
    "has_speech": {"type": "boolean"},
    "speaker_count": {"type": "number", "range": [0, 5], "step": 1}
}

audio_decision = model.system_one(media={"audio": audio_clip}, questions=questions_audio)
print(audio_decision)

6. System 2: Generative Chat & Reasoning

input_tokens = torch.tensor([[101, 2054, 2003, 1037, 102]], device=device)
generated_tokens = model.generate(input_tokens, max_new_tokens=32)
print("Generated Tokens:", generated_tokens.tolist())

---

📊 Hardware & Resource Requirements

---

📜 Citation & Credits

Special thanks to Liquid AI for training and open-sourcing the base foundation models:

• LiquidAI/LFM3-Omni-2B
• LiquidAI/d1-Omni-2B

@misc{lfm3_omni_dualhead_2026,
  author = {JonusNattapong},
  title = {LFM3-Omni Dual-Head Architecture: System 1 Decision and System 2 Generation on a Shared Backbone},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/JonusNattapong/lfm3-omni-dualhead}}
}

---

📄 License

This project is released under the Apache 2.0 License.