AI & Tech··1 sources verified
Model Annealing Trains Small-Data Networks by Reviewing Their Own Past Selves
Summarised by Relevant News AI · Read time: 3 min
Researchers propose model annealing, a training method that repeatedly reheats and cools the learning rate, keeps a memory of past cooled states, and uses adversarial examples where the student and memory disagree. Tested on CIFAR-100, Tiny-ImageNet and Imagewoof across four architecture families, it beat cross-entropy training in 28 of 33 configurations, improving top-1 accuracy by 1.3 points on average and matching snapshot distillation. The method also improved likelihood, calibration and robustness, and it adds no inference cost because the deployed network is the final student.
Why it matters: Teams training models on limited labelled data may gain better accuracy, calibration and robustness without extra compute at inference time.
All sources