PML-22L-O -- Password Modeling Language

Part of the Password Modeling Language (PML) series alongside PML-6L and PML-12L-O.

This model is a separate architecture trained from scratch (not a finetune of PML-6L or PML-12L-O) on 57 million USER+COUNTRY→password pairs extracted from stealer logs. PML-22L-O is the deep variant (22 layers, 384-dim embeddings) — following the MobileLLM hypothesis that deeper-and-thinner architectures outperform wider-and-shallower ones at the same parameter budget.

Its sibling PML-12L-O is the wide variant (12 layers, 512-dim embeddings). Both share approximately 46M parameters, enabling a controlled deep-versus-wide comparison.

Architecture

Parameter	Value
Layers	22
Hidden dimension	384
Attention heads	6
Head dimension	64
FFN hidden dimension	1024 (8/3 × 384)
Total parameters	45,237,120
Vocabulary size	8192 (BBPE)
Max sequence length	48
Position encoding	RoPE
Activation	SwiGLU
Normalization	RMSNorm
Attention	Scaled Dot-Product (PyTorch SDPA)
Training precision	FP16

Conditioning Tags

[USER:john_doe][COUNTRY:TR][LEN:12][CHARS:LOWER+UPPER+DIGIT]:myPassword1

Five augmentation modes during training: 40% full tag, 20% user-only, 15% context, 15% unconditional, 10% constraint-only.

Loss Masking

Only the password portion (after ]:) contributes to loss. Tag tokens masked with −100. EOS weighted 5× to encourage complete generation.

Training Data

57 M unique USER+COUNTRY→password pairs from 200+ GB of publicly available stealer log archives, filtered through a strict gold-quality pipeline:

Filter	Rule
Unicode letter	Username must contain ≥1 letter
$ sign	Reject if present in username
Digit ratio	≥8 chars + >60% digits → reject
File extensions	`.pdf`, `.exe`, `.html`, …
URL paths	≥2 slashes → reject
Generic names	`null`, `none`, `undefined`, `n/a`, …
Template markers	`xxx` patterns
Length	User 2–64, password ≥4 chars

Training Configuration

Hyperparameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.95)
Peak LR / min LR	2×10⁻⁴ → 2×10⁻⁵ (cosine, 70 % schedule)
Warmup steps	5 000
Label smoothing	0.05
Batch size	256 (eff. 2 048 with 8× grad accum)
Weight decay	0.1 (projections), 0.0 (biases, norms, embedding)
Gradient clipping	1.0
EOS weight	5×
Hardware	RTX 4060 Laptop (8 GB VRAM)
Training time	~13 h/epoch

Loss Progression

Epoch	Train Loss	Val Loss	Val Perplexity
1	5.717	5.150	172.5
2	5.337	5.079	160.5
3	5.284	5.042	154.8
4	5.225	5.016	150.8

Deep vs. Wide: PML-22L-O vs. PML-12L-O

Metric	PML-12L-O (wide)	PML-22L-O (deep)
Architecture	12L-512E-8H	22L-384E-6H
Parameters	46.1M	45.2M
Val Loss (Ep4)	5.006	5.016
Cultural patterns by Ep1	username-only	username + 1907
Cover @1000 guesses	1.40 %	1.65 %
Speed (beam=40, FP16)	1013 pw/s	990 pw/s

Key finding: the deeper model learns culturally-aware password patterns 3× faster than the wider model (produces fb1907 by epoch 1 vs. epoch 2–3 for the 12L variant). Cover rate is also 18 % higher at the 1 000‑guess budget.

Generation Speed

Benchmarked on RTX 4060 with FP16 autocast:

Strategy	Speed
`generate_fast()` (batch=256)	1386 pw/s
`generate_beam()` (beam=40)	990 pw/s
`generate()` (single)	5–10 pw/s

Cover Rate

2 000 random validation samples, 1 000 beam-search guesses per credential:

Budget	Cover
@1 guess	0.10 %
@10 guesses	0.55 %
@100 guesses	1.30 %
@1 000 guesses	1.65 %

Usage

Fast generation

import torch
from model_v5 import PasswordLLaMA
from tokenizers import Tokenizer

model = PasswordLLaMA(vocab_size=8192, n_layer=22, n_embd=384, n_head=6, max_seq_len=48)
state = torch.load("model.safetensors", weights_only=True)
model.load_state_dict(state, strict=True)
model.eval()

toker = Tokenizer.from_file("tokenizer.json")
prefix_ids = toker.encode("[USER:john][COUNTRY:TR]:").ids

with torch.no_grad():
    pws = model.generate_fast(toker, temperature=0.8, top_k=50,
                               max_len=48, min_len=4,
                               prefix_ids=prefix_ids, batch_size=256, device="cuda")
print(pws[:5])

Beam search (higher quality)

pws = model.generate_beam(toker, temperature=0.8, top_k=50,
                           max_len=48, min_len=6,
                           prefix_ids=prefix_ids,
                           n_passwords=100, beam=40,
                           length_penalty=0.85, device="cuda")

With Transformers pipeline

from transformers import pipeline
pipe = pipeline("text-generation", model="K0D3IN/PML-22L-O")
result = pipe("[USER:john][COUNTRY:TR]:", max_new_tokens=16)

Intended Use

✅ Appropriate	❌ Inappropriate
Authorized penetration testing	Credential stuffing against live systems
Red team engagements with written authorization	Account takeover
Password policy research	Illegal access to systems
Security awareness training	Privacy violations
Academic password research	Harassment or doxxing
Internal security audits	Any use without explicit authorization

Ethical Notice

This model generates password candidates that may match real user credentials.

Passwords generated are statistical patterns derived from stealer log archives. They do not contain actual training examples. However, due to the nature of password generation, some outputs may coincidentally match real passwords currently in use.

Users are solely responsible for:

Obtaining proper authorization before any security testing
Complying with all applicable laws (CFAA, GDPR, KVKK, etc.)
Ensuring testing is conducted within authorized scope only
Secure handling and disposal of generated password lists

This model should NOT be used for:

Any illegal activity
Attacking systems without explicit written permission
Mass credential stuffing or account takeover attempts
Harassing individuals or organizations

Files

File	Size	Description
`model.safetensors`	~181 MB	Model weights
`config.json`	~1 KB	Model configuration
`tokenizer.json`	~2 MB	BBPE tokenizer (vocab=8192)
`model_v5.py`	~12 KB	Model definition
`demo/`	—	Local Gradio app (run with `python demo/app.py`)

Citation

@software{PML-22L-O,
  author = {Mübeşşir Yusuf Akhan},
  title = {PML-22L-O: Conditional Password Generation via USER+COUNTRY},
  year = {2026},
  url = {https://huggingface.co/K0D3IN/PML-22L-O}
}

License

This project is licensed under the MIT License — see the LICENSE file for details.

Acknowledgments

This model is part of the PML series alongside PML-6L and {sibling}.

To my mother, who has always supported my ideas
To my father, who bought me the computer that made this project possible
To my friend Özüm, who supports me in realizing my ideals

Support Open Source AI Research

Every donation helps maintain and improve this project:

Monero (XMR):

83iqXtvVu28ZiL9bsATMerSgbFFiD1J1jc96CcxJLEnAW3KBmBKedWnUAeLvLvEA9aBiUBpHQJs1iNHYtkTLZbNUEymobSS

Bitcoin (BTC):

bc1qmnlvpukcgl0hsr7nje0x8555mhtxjt80wtmlxm