← Back to Architecture

residual mixing

Architecture
Used in
7 PRs
Best BPB
1.0577
Avg BPB
1.2732

Submissions

PR #103by MatthewHRockwell
1.5000
PR #443by CREVIOS
1.1431
PR #460by abhishekrajdhar
1.2928
PR #790by danialht
1.1172
PR #1180by estesryan
1.0577
PR #1527by alphastar1111
1.2026
PR #2159by visin109
1.5990

Hyperparameters Across PRs

pr_numberparameters
103—
443—
460—
790{"layers":11,"dimensions":512,"mlp_multiplier":3.5,"mha":"8/8","bigramhash":8192,"xsa":"all layers"}
1180—
1527—
2159—