← Back to Architecture

Soft MoE

Architecture
Used in
1 PRs
Best BPB
1.1826
Avg BPB
1.1826

Submissions

PR #660by HugoOchoaLP
1.1826

Hyperparameters Across PRs

pr_numberparameters
660{"num_experts":2,"moe_layers":"last 2 layers","moe_start_layer":8}