← Back to Architecture

MoE

Architecture
Used in
6 PRs
Best BPB
0.8335
Avg BPB
1.1266

Submissions

PR #480by imyesung
1.1456
PR #981by BurguerJohn
1.4893
PR #1451by davie2009kh
1.1180
PR #1901by Karen042009
0.8335
PR #2102by MaxIv25
1.1092
PR #2107by TidalTunes
1.0643

Hyperparameters Across PRs

pr_numberparameters
480{"experts":2,"expert_multiplier":1.5}
981{"moe_layers":0,"total_layers":2}
1451—
1901{"shared_experts":1,"specialized_experts":3,"top_k":1}
2102{"layers":[4,5],"experts":2,"routing":"top-1","enable_at":0.3}
2107—