Ministral-3-3B-Instruct-2512 Heretic Adaptive v1

This repository contains two BF16 derivatives of mistralai/Ministral-3-3B-Instruct-2512, created with dborzoff/heretic. The adaptive fork preserves Heretic's projection-based weight editing while adding broader search, exact trial restoration, multi-fidelity perplexity measurement, and reproducible experiment artifacts.

The two variants represent different operating points rather than a single universal winner:

• balanced/ is the recommended default when preserving the base model is the priority. Its exact post-export perplexity change was +0.00380%.
• max/ is the more aggressive variant. It produced fewer policy refusals and evasive answers, with an exact perplexity change of +0.04696%.

Both variants are full BF16 Transformers checkpoints stored in subfolders of this repository.

Measured trade-off

The semantic evaluation was performed blind on 136 multilingual prompts. Each system received the same prompts and generation limit of up to 2,048 new tokens. Perplexity was then measured independently on 400 windows of 512 tokens using the fully exported checkpoints.

Definitions used in the table:

• Delivered combines complete compliance, degraded-but-relevant compliance, and responses truncated after providing relevant content.
• Evasion combines substitution, legal redirection, and protective inversion: the model responds, but does not provide the requested content.
• Refusal + evasion is the residual behavior targeted by this study.
• Incoherent failure is reported separately and is not counted as a safety refusal.

The full text-free labels, aggregate summaries, and exact perplexity report are published in evaluation/. They contain identifiers, labels, confidence values, and metrics, but no prompt or response text.

Which variant should I use?

The difference in exact perplexity between the two edited variants is small, but it is not the only quality signal. The semantic evaluation shows a real trade-off, so downstream capability tests should determine the final choice.

Loading with Transformers

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "DmitryDB/Ministral-3-3B-Instruct-2512-Heretic-Adaptive-v1"
variant = "balanced"  # Change to "max" for the more aggressive variant.

processor = AutoProcessor.from_pretrained(repo, subfolder=variant)
model = AutoModelForImageTextToText.from_pretrained(
    repo,
    subfolder=variant,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Each subfolder includes its own configuration, processor/tokenizer files, and weight index. The model class remains AutoModelForImageTextToText, matching the base checkpoint.

Search procedure

The published variants were selected from one 600-trial Optuna journal. The search was deliberately split into broad exploration followed by shared adaptive refinement:

1. Trials 1–60: seeded random exploration.
2. Trials 61–120: multivariate TPE refinement of the random history.
3. Trials 121–180: an independent scrambled-Sobol exploration.
4. Trials 181–240: multivariate TPE refinement of the Sobol history.
5. Trials 241–600: one shared multivariate TPE study after merging both histories.

The animation uses real journal measurements. During search, each point is positioned by a keyword refusal proxy and a short perplexity surrogate. The static figure above uses the slower post-export semantic evaluation and exact 400-by-512 perplexity measurement. These are intentionally different stages: the inexpensive proxy guides exploration, while the final measurements choose and document the released checkpoints.

The dry numeric plot data for all 600 trials is available as evaluation/search_trials.json. It contains no prompts or model responses.

What the adaptive fork changes

Compared with the upstream Heretic workflow used as the starting point, this study adds or repairs:

• architecture-aware discovery of editable attention, dense MLP, shared expert, and fused routed-expert output projections;
• correct handling of three-dimensional fused MoE expert tensors;
• component-specific search bounds derived from the discovered topology;
• widened search ranges instead of relying on one fixed depth profile;
• GPU perplexity evaluation with a cheaper search-time surrogate and a larger exact finalist measurement;
• deterministic random and scrambled-Sobol exploration before multivariate TPE refinement;
• one journal shared by parallel workers, with exact trial budgets and Windows locking;
• restoration and export of any completed trial directly from its stored parameters;
• separation of optimization metrics, diagnostic probes, and blind final semantic evaluation.

The method still edits output projection weights along estimated refusal directions. Adaptive search changes where and how strongly those edits are applied; it does not retrain the base model or add an inference-time adapter.

For implementation details, configuration controls, tests, and known gaps, see the repository's FORK.md.

Evaluation artifacts and provenance

The checkpoints were rebuilt from their exact trial records, exported as full weights, reloaded, and then measured. The final semantic comparison was blind: the evaluator did not receive the model names, trial numbers, journal metrics, or perplexity values.

Limitations

• The semantic evaluation contains 136 prompts. It is useful for comparing these three systems under one protocol, but it is not a comprehensive safety or capability benchmark.
• The benchmark is multilingual, but the per-language sample sizes are too small to support strong language-specific claims.
• Semantic labels can contain evaluator error. Row-level text-free labels are included so counts and mappings remain auditable without redistributing the evaluation text.
• Perplexity is a broad distribution-shift diagnostic. A low change does not prove that reasoning, instruction following, factuality, vision performance, or every downstream capability was preserved.
• Search-time perplexity was a lower-cost surrogate. Only the released finalists received the larger exact measurement, so surrogate values should not be compared directly with the final percentages.
• These edits reduce refusal behavior. They do not make model outputs correct, lawful, harmless, or appropriate for deployment.

Intended use and responsibility

This release is intended for research into refusal behavior, weight editing, evaluation methodology, and the quality cost of model modification. Users are responsible for evaluating the selected variant in their own environment and for applying suitable access controls and safeguards.

The base model and these derivative weights are distributed under Apache-2.0.

Links

• Adaptive Heretic source code
• Detailed fork documentation
• Base model