English | 简体中文
MiniMind Dense 64M: One-Epoch Mini SFT
Research checkpoint release, updated 2026-11-01. It is an educational research checkpoint, not a reliable assistant. Validation NLL and some stopping behavior improved, but task completion did not improve over its own pretrained baseline on the fixed diagnostic set. Scores are assistant-authored initial judgments, with one unresolved case; no independent review is claimed.
1. Model Overview
This is a full-parameter supervised fine-tune of our own MiniMind mini pretrained model, trained and operated by yanpeng with AI assistance. Its pretrained weights were trained from random initialization, not initialized from official MiniMind model weights. The architecture and tokenizer reference MiniMind; this experiment does not reproduce the complete official model or inherit its reported capabilities.
The experiment asks whether assistant-only supervision can turn the existing continuation model into a better assistant under fixed conditions. It completed the engineering workflow, but did not meet the task-improvement objective. This card preserves that negative result alongside measurable partial changes.
| Item | Actual Model |
|---|---|
| Type | Dense decoder-only causal language model |
| Unique parameters | 63,912,192, approximately 63.91M |
| Architecture | 8 layers; hidden size 768; FFN intermediate size 2,432 |
| Attention | 8 query heads, 4 KV heads; Q/K RMSNorm; RoPE base 1,000,000 |
| Tokenizer | MiniMind, vocabulary 6,400; BOS=1, EOS=2, PAD=0 |
| Tied weights | Embedding and LM head share one Parameter |
| SFT initialization | Our own pretraining final step 4,943; fresh SFT AdamW/scheduler |
| Stages completed | Our mini pretraining, then this one-epoch SFT; no DPO or RL |
| Evaluated S1 | Final SFT step 49,087; not selected by comparing checkpoint scores |
| Export | Native FP32 safetensors; tensor file 255,659,304 bytes, approximately 255.66 MB |
| Interface | Local Python runtime; not a Transformers AutoModel adapter |
| Context scope | Trained and evaluated within 768 tokens; longer-context quality untested |
The upstream architecture/tokenizer reference is MiniMind commit f659b557. The pretraining architecture reference and SFT training-recipe choices are provenance, not claims of an exact official end-to-end reproduction.
2. Data and Supervision
2.1 Data Version
| Item | Actual Data |
|---|---|
| Upstream source | jingyaogong/minimind_dataset |
| Pinned revision | 312afb4f76391145c6902f765bb51691c09a12f5 |
| Source file | sft_t2t_mini.jsonl, 905,718 records |
| Accepted conversations | 787,425 |
| Train / validation | 785,377 / 2,048 |
| Limited revisions | 14 identity introductions and 2 specific content corrections |
| Final preparation holds | 21,073; some are unresolved contexts, not proven errors |
| Post-revision duplicates | 15 set aside |
| Train / validation supervised targets | 326,249,045 / 833,052, counted after the causal shift |
Preparation checked structure, declared-tool protocols and parameter schemas, real template encoding, length and valid targets. Earlier stages excluded overlength conversations, exact duplicates, two confirmed bad records, and undeclared calls. The final preparation applied bounded revisions/holds, re-encoded, deduplicated and split; these counts refer to different stages and should not be subtracted independently from the original total.
Split groups are connected components sharing any normalized user message or normalized full conversation. Normalization uses NFKC, case folding and collapsed whitespace for grouping only. Seed 20261031 selects whole groups within the validation budget; development/review examples are protected on the training side. Exact dialogue and defined normalized-question overlap across splits was zero. This is not semantic near-duplicate or external benchmark contamination clearance, and the validation set is not an independent blind test.
2.2 Encoding and Identity Policy
Complete encoded conversations longer than 768 are excluded, not truncated. The template is rendered once and tokenized without adding special tokens again. Right padding reaches 768. Assistant bodies, retained thinking, call text and the complete message-ending sequence are supervised; role headers, user/system text, tool results and padding are not targets at those positions. Context is still visible to the model; labels shift only in the causal loss.
The 14 bounded identity revisions identify yanpeng as the trainer and credit MiniMind's architecture/project; they do not globally replace people or project names. Their obsolete reasoning is cleared. This is a data policy, not evidence that the model reliably recalls its identity; the post-hoc identity probes below failed.
Content truth was reviewed only in bounded samples. Anonymous bundled tool results have limited structural correspondence guarantees. Semantic overlap, comprehensive factual quality, safety, personal-information leakage and full source-rights review were not cleared. Raw and derived training JSONL files are not included in this inference release.
3. Training Recipe and Cost
| Item | Actual Configuration |
|---|---|
| Hardware | One NVIDIA RTX 4060 Ti, 8 GB VRAM |
| Training system | Project's own PyTorch loop; not TRL Trainer |
| Parameter / compute precision | FP32 parameter state; CUDA BF16 autocast |
| Updated parameters | All unique trainable parameters, one AdamW group |
| Micro-batch / accumulation | 4 / 4; usually 16 conversations per update |
| Loss | Causal cross-entropy; loss sum divided by all valid assistant targets in the current accumulation window |
| AdamW | lr=1e-5; betas=(0.9,0.999); eps=1e-8; weight decay=0.01; foreach/fused=False |
| Schedule | No warmup; optimizer-update-indexed cosine decay to 1e-6 |
| Clipping | Global gradient norm limit 1.0 after token normalization; logged norm is pre-clip |
| Seeds / data workers | seed=42, sampler_seed=42; num_workers=0 |
| Training amount | One epoch, 49,087 updates and 196,345 micro-batches |
| Final update | One conversation, one micro-batch, 413 valid targets; not dropped |
| Validation | Full 2,048 records initially, every 5,000 updates and finally; BF16 monitoring |
| Checkpoints | Every 1,000 updates and completion; 50 full recovery checkpoints |
| Stop policy | 14-hour cumulative active-run budget with boundary save/pause; not reached |
Training ran from 2026-10-31 17:43:07 to 2026-11-01 04:26:24, Asia/Shanghai (UTC+08:00), and exited successfully. Wall time was approximately 10 hours 43 minutes 17 seconds: update windows approximately 10h12m24s, checkpoint saving/receipts 22m46s, validation 6m53s, and other startup/orchestration overhead approximately 1m13s. No extra rental or paid evaluator was used for this run; electricity and monetary operating cost were not measured.
The largest logged update peak allocated memory was approximately 3.77 GiB; post-update allocated memory was approximately 0.789 GiB and reserved memory ranged approximately 3.37–3.88 GiB. These are sampled allocator readings, not continuous whole-device measurements.
The final code passed a same-device continuous-20 versus 10-plus-resume-to-20 GPU gate under the full schedule. The final run's update/sample/target totals, tail, scheduler, AdamW state, sampler order, finite tensors and final checkpoint bytes were audited. All 50 checkpoint/receipt pairs were inventoried; not every historical checkpoint was fully hashed and loaded. This is not a guarantee of cross-device bitwise reproducibility.
4. Evaluation and Findings
4.1 Fixed Paired Diagnostic
B0 is our own pretrained model; S1 is the final SFT model. The 32 core cases cover everyday questions, instruction constraints, supplied material and fixed multi-turn contexts, eight each. Four tool cases are supplementary and do not execute real services.
Both use identical actual prompt text/token IDs, system, fixed histories, tools, tokenizer, template and an empty prefilled think region. Generation is CUDA FP32, eval/inference mode, greedy, batch 1, at most 256 new tokens, total context budget 768, EOS=2; TF32/autocast are disabled. NLL uses all 2,048 validation records, batch 4, and 833,052 assistant targets. Scoring is the original per-case checklist plus the same pre-SFT-approved V2 location-only patch.
| Metric | B0 | S1 |
|---|---|---|
| Full FP32 validation NLL | 2.3954561617 | 1.7210819774 |
| Core task judgments | 2 pass, 30 fail | 0 confirmed pass, 31 fail, 1 uncertain |
| Confirmed core pass rate | 6.25% | 0%; upper bound 3.125% if the uncertain case passes |
| Supplementary tool judgments | 0/4 pass | 0/4 pass |
| EOS / length-limit stops | 0 / 36 | 17 / 19 |
| Mean / median new tokens | 256 / 256 | 155.19 / 256 |
NLL decreased by approximately 28.15%. It measures prediction of real assistant targets under supplied context, not whole-answer correctness in free generation. These are SFT validation numbers; they are not comparable to the different validation set and full-text objective in the pretraining model card.
Core pairing contains 30 both-nonpass cases, one confirmed regression, and one uncertain pair, with no confirmed positive task flip. The uncertain case copies the correct key-location context but does not clearly answer the final location; it remains unresolved. Even counting it as a pass does not exceed B0's two passes.
All semantic judgments are assistant-authored initial reviews, not independently or exhaustively reviewed by the user. Hash, coverage, quote and aggregation checks do not certify semantic correctness. This small, already inspected diagnostic set is not a public benchmark or blind test.
4.2 Partial Changes and Failures
A location prompt requests only the location from a supplied record. S1 outputs:
青禾图书角周三开放，开始时间14:20，结束时间17:10，地点是二楼东侧。
It preserves the given times and stops with EOS, whereas B0 repeated text with an incorrect start time. It still fails the location-only instruction because it adds dates/times. This is a partial change, not a task pass.
For a radio-repair summary, S1 preserves the person, battery replacement and working-again outcome, but copies the two-sentence source instead of producing the requested one-sentence summary. Some other outputs merely echo the question and then end.
Tool-related risks remain: using lookup_book for weather, malformed call JSON, and inventing humidity/wind absent from a supplied weather result. No tool was actually executed. Reduced repetition or more EOS stops must not be read as reliable instruction following, factuality or safe tool use.
4.3 Post-Hoc Identity Diagnostic
After the main evaluation, three additional prompts ask who the model is, who trained it, and whether pretraining began randomly or continued official weights. Under the same S1 FP32 greedy settings, all three reach the 256-token limit without a correct source attribution; none identifies yanpeng/MiniMind as intended. One claims that it has no actual training data. The third question supplies two alternatives and is not an unprompted-recall test.
These are post-hoc diagnostics, not added to the 36 main cases, not blind, and not paired against B0. Trainer/source facts in this card come from experiment records, not the model's self-description.
5. Local Loading
The published inference package contains model.safetensors, config.json, tokenizer/, runtime/, load_model.py, requirements.txt, the two cards and chat.py.
Download this repository with the Hugging Face CLI (hf download yanpeng051821/minimind-dense-64m-mini-sft --local-dir minimind-mini-sft), install dependencies from requirements.txt, and run from the downloaded directory:
python -I -B chat.py --prompt "你好，请简单介绍一下你自己。" --max-new-tokens 256
The example is a way to inspect behavior, not a promise of a correct introduction. chat.py defaults to CPU FP32, uses the same neutral Chinese assistant system as evaluation, renders the packaged chat template with add_generation_prompt=True and open_thinking=False, and tokenizes with add_special_tokens=False. It generates one reply, executes no tools, performs no sampling/retries, and rejects prompt-plus-budget beyond 768 without silent truncation. CPU performance/outputs are not asserted bitwise identical to the CUDA evaluation.
The original load_model.py --prompt interface remains a raw continuation/loading check; it does not add the SFT chat template. Do not substitute AutoModel.from_pretrained or assume automatic support by Hub inference providers.
Recorded dependencies are PyTorch 2.14.0, Transformers 4.57.6, tokenizers 0.22.2 and safetensors 0.8.0. Select the appropriate PyTorch build for the platform; only the existing local environment has been tested, not a clean installation elsewhere. Review the included custom Python before execution; file hashes are not signatures or a code sandbox.
6. Export and Reproduction
The final checkpoint was strictly loaded into the matching architecture and exported without changing its FP32 logical parameters. A fresh offline CPU process reconstructed the model with the package's runtime/config/tokenizer. The full state digest and tied Parameter matched; inputs, masks, logits for two short prefixes and a short greedy continuation were exactly equal, with zero maximum logit difference. This is a bounded export check, not all-input or cross-device equivalence.
| Identity | SHA256 |
|---|---|
| Final SFT checkpoint | 181ce5bbc64c0c6ad0f3b79682f71df33206665c8212182422ce1363fec1e03a |
| model.safetensors | 4fdc2b4d4f2bbc1f85ca0a7a68eb615fea96cfaf3c7ddcf9cb9c8e03dcb995b5 |
| Logical model-state digest | 733c18c9775eadd7f67330fabb0568ab61103f9d5fba995e6d967405c904cc62 |
| Initial B0 state digest | 8d10d76090b2714c055a95123c3583c6941b1ecb6af84a10f0f68dbe21cb783f |
| SFT training contract | 3ac716dee78ce33d5d418e7f1f5e7e6b176dcdad6d06c52ac608322172c1abe9 |
| SFT data contract | 55bea98df2af3f6105fc03e06711b2602ba38e678b412720dcfff928e9f1da4d |
The source project is AI_research, with this experiment under training_engineering/case_studies/minimind/experiments/sft_mini_v1. The inference runtime, bilingual cards, recipes and evaluation evidence are included in this Hub repository. The latest SFT training integration and reports have not yet been pushed to GitHub; the GitHub root link does not assert that those uncommitted changes are accessible there. File identities are listed in export_manifest.json; recipes/ contains path-redacted historical snapshots and evidence/ retains diagnostic inputs, generated token IDs, text and judgments.
Changing documentation or adding chat.py requires a new release manifest; the evaluated native bundle and its existing manifest remain unchanged. Full recovery checkpoints retain optimizer/scheduler/sampler/RNG state locally and are not replaced by this inference package.
7. Intended Use and Limitations
This checkpoint is useful for learning, inspecting training effects and reproducing a bounded negative-result experiment. It is not a production assistant, trusted knowledge source, verified identity responder or autonomous tool agent. Strong repetition, question/source copying, missed constraints, unsupported claims and tool misuse were observed.
Long-context quality beyond 768, English proficiency, systematic safety/bias, privacy leakage and memorization have not been established. The language of this English card is not evidence of English generation ability. No extra training, checkpoint selection, changed decoding or relaxed rubric was used to manufacture an improvement after seeing the result.
Core components were implemented by the learner with guidance; delegated integration, audit, operations, evaluation, export and documentation were assisted by an AI coding agent. The project is not represented as entirely independently implemented by the learner.
8. Sources and License Status
MiniMind's pinned source license and tokenizer provenance are credited separately from this model's weights. The pinned dataset card declares Apache-2.0 and CC-BY-NC-2.0; the existing review has not established a complete per-file source-license mapping or the full weight-redistribution scope.
No blanket model-weight/package license or unconditional commercial rights are asserted, and no license tag has been selected automatically. Research or noncommercial labeling alone does not resolve missing permissions. Applicable notices are preserved in THIRD_PARTY_NOTICES.md and licenses/; public redistribution scope and original-contribution terms still require clarification; this card records the uncertainty rather than giving legal advice.
Release scope: inference package and bounded experiment evidence, authorized by the trainer on 2026-11-01. Full recovery checkpoints and raw/derived training data are not included. Publication does not resolve the license uncertainties above. Use a pinned Hub revision and verify export_manifest.json; download-verification results are recorded separately by the publisher.