Vortex-50M (Cybersecurity & Cryptography Specialist)

A 49,846,528-parameter instruction-tuned decoder-only Transformer, built from first principles on torch.nn and fine-tuned for specialized cybersecurity, network defense, threat analysis, and cryptographic protocols.

49,846,528 params  ·  512d × 18L  ·  8Q/2KV GQA  ·  vocab 16,387 (tied)
17% embedding  /  83% transformer blocks  ·  Strictly within 50M parameter budget

---

Benchmark Results

Evaluated directly via EleutherAI's lm-evaluation-harness across 4 standard multiple-choice and reasoning tasks using length normalization where applicable:

Evaluation Methodology: All benchmarks are scored using src/eval_competition.py with the official EleutherAI harness. No hosted inference APIs were touched during training or evaluation.

---

Parameter Accounting & Budget

The parameter budget is verified from the live module tree using src/param_count.py:

==================================================================
  vortex-50m
==================================================================
  hidden                 512
  layers                 18
  heads                  8Q / 2KV
  head_dim               64
  intermediate           1072
  context                2048
  vocab                  16,387 (tied=True)
------------------------------------------------------------------
  TRAINABLE PARAMS       49,846,528
  budget                 50,000,000
  verdict                [PASS]  49.847M  (99.69% of budget)
------------------------------------------------------------------
  embedding                 8,390,144   16.8%
  attention                11,798,784   23.7%
  mlp                      29,638,656   59.5%
  norm                            512    0.0%
  ---                             ---
  in transformer blocks    41,456,384   83.2%
==================================================================

lm_head is tied to the input embedding table, ensuring maximum capacity is directed into the 18 transformer layers (83.2% of weights inside transformer blocks).

---

Architectural Principles at 50M Parameters

• Depth over Width via Compact Vocab: A custom 16,384 BPE vocabulary costs only ~8.4M parameters in the embedding table (17% of budget). In contrast, standard 32K or 150K vocabularies consume 35% to 150%+ of a 50M budget before a single layer can be placed. This allows Vortex-50M to run 18 deep layers at 512 hidden dimension.
• Grouped-Query Attention (GQA 8/2): 8 query heads and 2 key/value heads provide a 4× KV cache reduction for fast on-device inference without sacrificing reasoning quality.
• QK-Norm (RMSNorm on Q and K): Applied per head before attention computation to prevent attention entropy collapse and numerical instability at small scales.
• SwiGLU MLP: Intermediate dimension of 1,072 (2.09× hidden dimension) with SwiGLU activation for superior non-linear representation capacity.
• ChatML Format & Special Control Tokens: Native support for <|im_start|> and <|im_end|> sequence markers with assistant-only loss masking during instruction tuning.

---

Quickstart & Inference

Vortex-50M runs natively with Hugging Face transformers using trust_remote_code=True:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "VTXAI/vortex-50m"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True
)

# Format prompt using ChatML
messages = [
    {
        "role": "system",
        "content": "You are Vortex-50M, an expert cybersecurity and cryptography assistant."
    },
    {
        "role": "user",
        "content": "Explain why AES-GCM is preferred over AES-CBC with HMAC in modern network protocols."
    }
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.3,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

---

Training & Fine-Tuning Pipeline

1. Pretraining: Pretrained from random initialization on 1 Billion tokens of curated high-quality web text (cosmopedia-v2 + fineweb-edu-dedup) using cosine decay down to 10% peak LR.
2. Supervised Instruction Tuning (SFT): Fine-tuned with ChatML masking on VTXAI/cyber-crypto-balanced-qa using AdamW (lr=1.8e-5, weight decay 0.01, cosine scheduler).
3. Loss Masking: User prompts and system instructions are masked out of the loss calculation; gradients are computed strictly on assistant generation tokens.

---

Verification & Integrity

The model passes all pre-flight and runtime consistency suites in src/:

• Parity Check (src/test_hf_modeling.py): Bit-exact agreement between custom torch.nn training code and Hugging Face AutoModelForCausalLM.
• Roundtrip Check (src/test_export_hf.py): Verified export, weight reload, and token generation consistency in clean environments.
• Budget Compliance (src/param_count.py): Verified 49,846,528 params ≤ 50,000,000 budget.