← Back to Architecture

attention modifications

Architecture
Used in
7 PRs
Best BPB
1.0590
Avg BPB
1.2747

Submissions

PR #981by BurguerJohn
1.4893
PR #1074by ldh-at
1.3288
PR #1168by gowtham0992
1.3639
PR #1558by Subramanyam6
1.4500
PR #2006by Elubrazione
1.0590
PR #2006by Elubrazione
1.0590
PR #2070by vardanbobo007
1.1730

Hyperparameters Across PRs

pr_numberparameters
981—
1074{"hyperbolic_qk_mix":0.02,"hyperbolic_radius_init":0.1}
1168—
1558{"qk_gain":4}
2006—
2006—
2070—