CIFAR-10 Self-Training Best Iterate (t=4)

Best self-trained iterate of an iterative self-training trajectory on CIFAR-10, reproducing the deep-learning analogue of Why Self-Training Helps and Hurts (arXiv:2602.14029, Appendix A).

Clean CIFAR-10 test accuracy: 46.76% (iteration t=4).

Trained on hard pseudo-labels produced by iterate t=3 on a fresh disjoint 5000-image subset. This is the risk minimum of the trajectory: denoising dominates up to this point (+4.3 points over the teacher).

Training configuration

• Architecture: ResNet-18 with CIFAR stem (3x3 conv, no maxpool), trained from scratch
• Iteration: t=4 of K=8
• Data: 5000 fresh disjoint CIFAR-10 train images per iteration (uoft-cs/cifar10)
• Teacher (t=0) labels: 40% symmetric noise; students: hard pseudo-labels from iterate t-1
• SGD lr 0.05, momentum 0.9, weight decay 5e-4, 5-epoch warmup + cosine, batch 128, 60 epochs
• Augmentation: random crop (pad 4) + horizontal flip
• torch 2.13.0+cu130, seed 104

Full trajectory, configs, metrics and plots: https://huggingface.co/datasets/pngwn/self-training-denoising-forgetting

Usage

import torch, torch.nn as nn, torchvision

def build_model():
    m = torchvision.models.resnet18(weights=None)
    m.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
    m.maxpool = nn.Identity()
    m.fc = nn.Linear(512, 10)
    return m

model = build_model()
sd = torch.load("model.pt", map_location="cpu", weights_only=True)
model.load_state_dict(sd)
model.eval()

Input: CIFAR-10 images normalized with mean (0.4914, 0.4822, 0.4465), std (0.2470, 0.2435, 0.2616).