← Back to Architecture

attention

Architecture
Used in
11 PRs
Best BPB
1.0543
Avg BPB
1.2052

Submissions

PR #579by newjordan
1.1355
PR #1088by serdardoesml
1.2542
PR #1219by Gusanidas
1.1084
PR #1354by samacqua
1.1092
PR #1355by mradassaad
1.1526
PR #1536by dexhunter
1.0775
PR #1540by aryanbhosale
1.0777
PR #1560by dexhunter
1.0741
PR #1842by Azura-Whiston
1.8838
PR #2000by edidiongumoh
1.3302
PR #2013by Wilbatronic
1.0543

Hyperparameters Across PRs

pr_numberparameters
579{"num_heads":10,"num_kv_heads":5,"XSA_layers":2}
1088{"layers":2}
1219{"layers":[2,4,6,8,10],"window_size":512}
1354—
1355{"layers":1,"position":4,"heads":"8/4"}
1536—
1540—
1560—
1842{"layers":12}
2000{"backend":"flash"}
2013{"layers":11}