Tiny Randomly Initialized
Base Model: VINAY-UMRETHE/Refusal-Classifier
Purpose
This is a tiny, randomly initialized model that has not been trained on any dataset, so do not use this model for any real refusal classification tasks. It is used for testing in the GitHub CI of p-e-w/heretic (implemented in feat: Generic BERT Classifier scorer plugin #500), similar to how the other reproduction tests use models from the tiny-random organization.
You can also use this model for debugging without loading the real models (e.g. VINAY-UMRETHE/Refusal-Classifier, VINAY-UMRETHE/Refusal-Classifier-V2, etc.), as they share the same architecture and configuration (ModernBERT).
How this model was created
from typing import cast
import torch
from transformers import (
AutoConfig,
AutoTokenizer,
ModernBertForSequenceClassification,
PreTrainedTokenizerBase,
set_seed,
)
source_model_id = "VINAY-UMRETHE/Refusal-Classifier"
save_folder = "tiny-refusal-classifier"
tokenizer = cast(
PreTrainedTokenizerBase,
AutoTokenizer.from_pretrained(source_model_id),
)
tokenizer.save_pretrained(save_folder)
config = AutoConfig.from_pretrained(source_model_id)
config.hidden_size = 32
config.intermediate_size = 128
config.num_attention_heads = 1
config.num_hidden_layers = 2
config.layer_types = config.layer_types[:2]
model = ModernBertForSequenceClassification(config)
# FP32, matching the source model's dtype.
# Expected: torch.float32
print(next(model.parameters()).dtype)
set_seed(75)
with torch.no_grad():
for name, parameter in sorted(model.named_parameters()):
torch.nn.init.normal_(parameter, 0, 0.5)
print(name, parameter.shape)
model.save_pretrained(save_folder)
# Upload:
# model.push_to_hub(repo_id="VINAY-UMRETHE/tiny-refusal-classifier")
# tokenizer.push_to_hub(repo_id="VINAY-UMRETHE/tiny-refusal-classifier")
License
Copyright © 2026 Vinay Umrethe.
This model is licensed under the Apache License 2.0 License.