val_bpb
1.4370
Architecture
Transformer
Optimizer
Muon
Artifact Size
15.15 MB
Training Techniques
Quantization
STE QAT
bits: 6
scope: all
Architecture
MLP3x
Expanded MLP hidden dimension to 3x the model dimension.
parameters: {"layers":11,"hidden_dim":1536}
SmearGate
Sigmoid-gated mechanism initialized near 0.95 to improve model behavior.
parameters: {"init":3}
BigramHash
Added a bigram hash feature with 4096 buckets and 128-dimensional embeddings.
parameters: {"buckets":4096,"dim":128}
tied embeddings
Input and output embeddings are tied.
parameters: null
Weight Averaging
SWA
parameters: {"every":50}
Optimizer
Muon
weight_decay: 0.04
momentum: null
other_params: {"matrix_lr":0.02,"scalar_lr":0.02,"embed_lr":0.03}
Compression
zstd
level: null
Evaluation
error correction table
parameters: {"use_correction":1,"boost_logit":20,"fixed_val_set":true}
Other
other
Built a compact position-to-token lookup table from worst predictions on the fixed validation set and applied it during evaluation to zero out loss on matched positions.
parameters: {"entries":907927,"table_size_bytes":2867053}
other
Used delta-encoded positions with varint encoding for compact correction-table storage.
parameters: {"avg_bytes_per_entry":3.16}
Novel Contributions
- Error Correction Table: a pre-computed position-to-token lookup table for the fixed validation set
- Delta-encoded positions plus varint encoding to compress correction entries efficiently
- On-the-fly correction table construction during evaluation without a separate build step
- Logit boosting for matched positions to achieve effectively zero loss on corrected tokens