← Back to Architecture

MLP4x

Architecture
Used in
15 PRs
Best BPB
1.0764
Avg BPB
1.1166

Submissions

PR #498by newjordan
1.1478
PR #842by JUSTSUJAY
1.3380
PR #1052by demouo
1.1978
PR #1260by dexhunter
1.0929
PR #1287by dentity007
1.1048
PR #1291by dentity007
1.0925
PR #1296by aryanbhosale
1.0926
PR #1326by aryanbhosale
1.0896
PR #1334by aryanbhosaleRECORD
1.0897
PR #1392by Its-Just-Crump
1.1020
PR #1423by aryanbhosale
1.0791
PR #1477by aryanbhosaleRECORD
1.0822
PR #1555by andrewbaggio1
1.0764
PR #1658by AVINASH0052
1.0810
PR #1747by swapp1990
1.0820

Hyperparameters Across PRs

pr_numberparameters
498{"multiplier":4,"hidden_size":2560}
842{"layers":5,"model_dim":512,"mlp_mult":4,"hidden":2048,"num_heads":8,"num_kv_heads":4}
1052—
1260{"multiplier":4}
1287{"multiplier":4}
1291{"multiplier":4}
1296—
1326—
1334{"vocab_size":4096}
1392{"multiplier":4}
1423{"multiplier":4}
1477—
1555{"multiplier":4}
1658{"multiplier":4,"hidden_dim":2048}
1747{"layers":11,"intermediate_dim":2048}