Learn how to fit a language model into 16MB. This site breaks down every technique used in OpenAI's Parameter Golf competition — from quantization and architecture tricks to test-time training — with interactive deep dives, real submission data, and code.

PRs Processed
1672
Best Record BPB
1.0567
Techniques
1182
Deep Dives
10
CURRENT RECORDPR #2135by codemath3000
val_bpb
1.0567
Architecture: Transformer
Optimizer: Muon
ArchitecturePartial RoPE, depth recurrence, XSA, SmearGate, Gated Attention
OptimizerMuon, Adam
Weight AveragingEMA
QuantizationGPTQ, int7, LQER
Test-Time TrainingLoRA TTT
Regularizationlogit softcap
Otherother
Sequence Lengthsequence_length
LR Schedulewarmdown
View PR details →

Deep Dives

quantization
Quantization Fundamentals
Reducing model size while preserving performance
5 sections
architecture modification
Architecture Tricks
U-Net skips, BigramHash, SmearGate, and more
9 sections
optimizer technique
The Muon Optimizer
Why Parameter Golf's best players abandoned Adam
11 sections
weight averaging
Weight Averaging
SWA, EMA, and ensemble-like approaches that cost almost nothing
12 sections
compression
Compression
zstd, pruning, and artifact size optimization
6 sections
test time training
Test-Time Training
Adapting models at inference time with LoRA, score-first TTT, and per-document fine-tuning
7 sections
lr schedule
Learning Rate Schedules
Warmdown, cosine, and schedule optimization
7 sections
initialization
Initialization
OrthoInit and weight initialization strategies
6 sections
regularization
Regularization
Weight decay, pruning, and overfitting prevention
7 sections
evaluation technique
Evaluation Strategies
Sliding window eval, N-gram mixing, and scoring techniques
8 sections

Record Progression

Record BPB (is_record)
All submissions
Baseline (1.9)

Technique Category Frequency

Top Submissions

View all 1672PRs →
#10.0000
PR #721
Middle-Out Compression: 0.0000 bpb (Shannon Limit Broken)
hypery11
#20.0000
PR #959
Record: Nacrith Log-Bias + Full-Rescore N-gram — val_bpb 0.00000035 (3-seed mean)
himanalot
#30.0109
PR #1076
Record: Packed Causal N-gram + Dirichlet Backoff — val_bpb 0.0109 (3-seed mean, NEW SOTA)
sofiabod
#40.0165
PR #943
Record: Compliance-First Packed Causal Memory + Dirichlet Mixing — val_bpb 0.01654407 (3-seed mean)
aamodbhatt
#50.0165
PR #944
Record: Compliance-First Packed Causal Memory + Dirichlet Mixing — val_bpb 0.01654407 (3-seed mean)
aamodbhatt
#60.0180
PR #1056
Record: Packed Causal N-gram + Dirichlet Backoff — val_bpb 0.0180 (3-seed mean)
sofiabod
#70.0214
PR #962
Record: 0.0214 bpb - Low Eval-Time Memory Regime: Packed Training N-gram Artifact + Learned Gate (No Phrase Cache)
AnirudhRahul
#80.0235
PR #1114
Record: Packed N-gram + Dirichlet CTW — val_bpb 0.0235 (1xB200)
minh-stakc
#90.0274
PR #945
Record: Order-16 Frozen N-gram Oracle + Learned Gate + TTT — val_bpb 0.0274 (3-seed mean)
TimPietrusky
#100.0280
PR #924
Order-16 Frozen N-gram Oracle + Score-First TTT (0.02801 BPB)
THUQiXuan