PR #2139
open[Record candidate] TTT Peer-LoRA Ensemble on PR #2014, val_bpb = 1.05749
by varunnealView on GitHub
val_bpb
1.0575
Architecture
Transformer
Optimizer
—
Artifact Size
~15.99 MB
Training Techniques
Test-Time Training
LoRA TTT
parameters: {"learning_rate":0.00015,"weight_decay":0.25}
Evaluation
sliding window eval
parameters: null
Regularization
weight decay
parameters: {"value":0.25}
Other
other
Peer-LoRA ensembling during test-time training evaluation, blending own and peer predictions on uncertain tokens using predictive entropy routing.
parameters: {"k":4,"threshold":0.5,"blend_weight_own":0.8}
other
Target-free confidence routing based on predictive entropy to decide when to use ensemble predictions.
parameters: {"threshold":0.5}
Novel Contributions
- Introduced peer-LoRA ensembling into the TTT evaluation loop
- Used target-free predictive-entropy routing to blend own and peer predictions
- Tuned per-doc LoRA learning rate and weight decay to improve the baseline
- Built on PR #2014 with doc-independent LoRAs