PML-12L-O -- Password Modeling Language

Part of the Password Modeling Language (PML) series alongside PML-6L and PML-22L-O.

This model is a separate architecture trained from scratch (not a finetune of PML-6L) on 57 million USER+COUNTRY→password pairs extracted from stealer logs. It conditions generation on both username and country tags, producing culturally-aware password candidates. PML-12L-O is the wide variant (12 layers, 512-dim embeddings).

Its sibling PML-22L-O is the deep variant (22 layers, 384-dim embeddings). Both share the same parameter budget (~46M), making them a controlled deep-versus-wide comparison.

Architecture

Parameter	Value
Layers	12
Hidden dimension	512
Attention heads	8
Head dimension	64
FFN hidden dimension	1365 (8/3 × 512)
Total parameters	46,144,000
Vocabulary size	8192 (BBPE)
Max sequence length	48
Position encoding	RoPE
Activation	SwiGLU
Normalization	RMSNorm
Attention	Scaled Dot-Product (PyTorch SDPA)
Training precision	FP16

Conditioning Tags

[USER:john_doe][COUNTRY:TR][LEN:12][CHARS:LOWER+UPPER+DIGIT]:myPassword1

Five augmentation modes during training: 40% full tag, 20% user-only, 15% context, 15% unconditional, 10% constraint-only.

Loss Masking

Only the password portion (after ]:) contributes to loss. Tag tokens masked with −100. EOS weighted 5× to encourage complete generation.

Training Data

57 M unique USER+COUNTRY→password pairs from 200+ GB of publicly available stealer log archives, filtered through a strict gold-quality pipeline:

Filter	Rule
Unicode letter	Username must contain ≥1 letter
$ sign	Reject if present in username
Digit ratio	≥8 chars + >60% digits → reject
File extensions	`.pdf`, `.exe`, `.html`, …
URL paths	≥2 slashes → reject
Generic names	`null`, `none`, `undefined`, `n/a`, …
Template markers	`xxx` patterns
Length	User 2–64, password ≥4 chars

Training Configuration

Hyperparameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.95)
Peak LR / min LR	2×10⁻⁴ → 2×10⁻⁵ (cosine, 70 % schedule, last 30 % at min)
Warmup steps	5 000
Label smoothing	0.05
Batch size	256 (eff. 2 048 with 8× grad accum)
Weight decay	0.1 (projections), 0.0 (biases, norms, embedding)
Gradient clipping	1.0
EOS weight	5×
Hardware	RTX 4060 Laptop (8 GB VRAM)
Training time	~13 h/epoch

Loss Progression

Epoch	Train Loss	Val Loss	Val Perplexity
1	5.717	5.124	168.0
2	5.337	5.060	157.7
3	5.284	5.027	152.5
4	5.225	5.006	150.3

Deep vs. Wide: PML-12L-O vs. PML-22L-O

Metric	PML-12L-O (wide)	PML-22L-O (deep)
Architecture	12L-512E-8H	22L-384E-6H
Parameters	46.1M	45.2M
Val Loss (Ep4)	5.006	5.016
Cover @1000 guesses	1.40 %	1.65 %
Speed (beam=40, FP16)	1013 pw/s	990 pw/s

The deeper model learns cultural patterns 3× faster and achieves higher cover rates, confirming the MobileLLM hypothesis for password generation.

Generation Speed

Benchmarked on RTX 4060 with FP16 autocast:

Strategy	Speed
`generate_fast()` (batch=384)	1593 pw/s
`generate_beam()` (beam=40)	1013 pw/s
`generate()` (single)	5–10 pw/s

Cover Rate

2 000 random validation samples, 1 000 beam-search guesses per credential:

Budget	Cover
@1 guess	0.10 %
@10 guesses	0.45 %
@100 guesses	1.05 %
@1 000 guesses	1.40 %

Usage

Fast generation

import torch
from model_v5 import PasswordLLaMA
from tokenizers import Tokenizer

model = PasswordLLaMA(vocab_size=8192, n_layer=12, n_embd=512, n_head=8, max_seq_len=48)
state = torch.load("model.safetensors", weights_only=True)
model.load_state_dict(state, strict=True)
model.eval()

toker = Tokenizer.from_file("tokenizer.json")
prefix_ids = toker.encode("[USER:john][COUNTRY:TR]:").ids

with torch.no_grad():
    pws = model.generate_fast(toker, temperature=0.8, top_k=50,
                               max_len=48, min_len=4,
                               prefix_ids=prefix_ids, batch_size=128, device="cuda")
print(pws[:5])

Beam search (higher quality)

pws = model.generate_beam(toker, temperature=0.8, top_k=50,
                           max_len=48, min_len=6,
                           prefix_ids=prefix_ids,
                           n_passwords=100, beam=40,
                           length_penalty=0.85, device="cuda")

With Transformers pipeline

from transformers import pipeline
pipe = pipeline("text-generation", model="K0D3IN/PML-12L-O")
result = pipe("[USER:john][COUNTRY:TR]:", max_new_tokens=16)

Intended Use

✅ Appropriate	❌ Inappropriate
Authorized penetration testing	Credential stuffing against live systems
Red team engagements with written authorization	Account takeover
Password policy research	Illegal access to systems
Security awareness training	Privacy violations
Academic password research	Harassment or doxxing
Internal security audits	Any use without explicit authorization

Ethical Notice

This model generates password candidates that may match real user credentials.

Passwords generated are statistical patterns derived from stealer log archives. They do not contain actual training examples. However, due to the nature of password generation, some outputs may coincidentally match real passwords currently in use.

Users are solely responsible for:

Obtaining proper authorization before any security testing
Complying with all applicable laws (CFAA, GDPR, KVKK, etc.)
Ensuring testing is conducted within authorized scope only
Secure handling and disposal of generated password lists

This model should NOT be used for:

Any illegal activity
Attacking systems without explicit written permission
Mass credential stuffing or account takeover attempts
Harassing individuals or organizations

Files

File	Size	Description
`model.safetensors`	~185 MB	Model weights
`config.json`	~1 KB	Model configuration
`tokenizer.json`	~2 MB	BBPE tokenizer (vocab=8192)
`model_v5.py`	~12 KB	Model definition
`demo/`	—	Local Gradio app (run with `python demo/app.py`)

Citation

@software{PML-12L-O,
  author = {Mübeşşir Yusuf Akhan},
  title = {PML-12L-O: Conditional Password Generation via USER+COUNTRY},
  year = {2026},
  url = {https://huggingface.co/K0D3IN/PML-12L-O}
}

License

This project is licensed under the MIT License — see the LICENSE file for details.

Acknowledgments

This model is part of the PML series alongside PML-6L and {sibling}.

To my mother, who has always supported my ideas
To my father, who bought me the computer that made this project possible
To my friend Özüm, who supports me in realizing my ideals

Support Open Source AI Research

Every donation helps maintain and improve this project:

Monero (XMR):

83iqXtvVu28ZiL9bsATMerSgbFFiD1J1jc96CcxJLEnAW3KBmBKedWnUAeLvLvEA9aBiUBpHQJs1iNHYtkTLZbNUEymobSS

Bitcoin (BTC):

bc1qmnlvpukcgl0hsr7nje0x8555mhtxjt80wtmlxm

K0D3IN/PML-12L-O