← Back to Architecture

MoE

Architecture
Used in
4 PRs
Best BPB
0.8335
Avg BPB
1.1466

Submissions

PR #480by imyesung
1.1456
PR #981by BurguerJohn
1.4893
PR #1451by davie2009kh
1.1180
PR #1901by Karen042009
0.8335

Hyperparameters Across PRs

pr_numberparameters
480{"experts":2,"expert_multiplier":1.5}
981{"moe_layers":0,"total_layers":2}
1451—
1901{"shared_experts":1,"specialized_experts":3,"top_k":1}