← Back to Architecture

SWA

Architecture
Used in
11 PRs
Best BPB
0.0180
Avg BPB
0.8987

Submissions

PR #484by Robby955
1.1185
PR #610by ChaosCodes
1.1190
PR #1030by sofiabod
0.1130
PR #1056by sofiabod
0.0180
PR #1512by Itssshikhar
1.1117
PR #1544by Abhishek8108
1.0283
PR #1545by Abhishek8108
1.0283
PR #1553by Abhishek8108
1.2097
PR #1632by Hkoyuer
1.0274
PR #1672by andrewbaggio1
1.0119
PR #1749by gracebml
1.0996

Hyperparameters Across PRs

pr_numberparameters
484—
610{"frequency_steps":50}
1030—
1056—
1512{"window_size":256,"full_attention_layers":5}
1544{"window":512,"heads":8,"kv_heads":4,"shared":true}
1545{"window":512,"heads":8,"kv_heads":4,"shared_weights":true}
1553{"layers":2,"window":512,"heads":8,"kv_heads":4,"shared_weights":true}
1632{"layers":2,"window":512,"heads":8,"kv_heads":4}
1672{"window":512,"qk_gain":5}
1749{"layers":2,"shared_weights":true,"window":512}