← Back to Architecture

attention modifications

Architecture
Used in
4 PRs
Best BPB
1.3288
Avg BPB
1.4080

Submissions

PR #981by BurguerJohn
1.4893
PR #1074by ldh-at
1.3288
PR #1168by gowtham0992
1.3639
PR #1558by Subramanyam6
1.4500

Hyperparameters Across PRs

pr_numberparameters
981—
1074{"hyperbolic_qk_mix":0.02,"hyperbolic_radius_init":0.1}
1168—
1558{"qk_gain":4}