← Back to Weight Averaging

SWA and EMA

Weight Averaging
Used in
1 PRs
Best BPB
1.1355
Avg BPB
1.1355

Submissions

PR #579by newjordan
1.1355

Hyperparameters Across PRs

pr_numberparameters
579{"SWA_frequency":"every 50 steps when scale < 0.2","EMA_decay":0.997,"EMA_applied_after_distillation":true}