PR #1021

Non-record: MC Dropout ensembling is negative for small LMs

Training Techniques

Novel Contributions

• Evaluated MC Dropout ensembling for a 17M-parameter language model
• Showed that averaging 16 dropout samples at inference does not improve BPB
• Found deterministic single-pass inference outperforms MC Dropout at both tested dropout rates
• Argued that dropout-induced sub-network diversity is insufficient at this scale