← Back to Architecture

attention

Architecture
Used in
8 PRs
Best BPB
1.0741
Avg BPB
1.1236

Submissions

PR #579by newjordan
1.1355
PR #1088by serdardoesml
1.2542
PR #1219by Gusanidas
1.1084
PR #1354by samacqua
1.1092
PR #1355by mradassaad
1.1526
PR #1536by dexhunter
1.0775
PR #1540by aryanbhosale
1.0777
PR #1560by dexhunter
1.0741

Hyperparameters Across PRs

pr_numberparameters
579{"num_heads":10,"num_kv_heads":5,"XSA_layers":2}
1088{"layers":2}
1219{"layers":[2,4,6,8,10],"window_size":512}
1354—
1355{"layers":1,"position":4,"heads":"8/4"}
1536—
1540—
1560—