← Back to Architecture

Gated DeltaNet

Architecture
Used in
8 PRs
Best BPB
1.0030
Avg BPB
1.1246

Submissions

PR #1370by Christopher-Lee-McClendon
1.0030
PR #1371by aarjunsrinivasan
1.4709
PR #1479by andrewbaggio1
1.1450
PR #1553by Abhishek8108
1.2097
PR #1562by joshkmartinez
1.0205
PR #1563by joshkmartinez
1.0205
PR #1632by Hkoyuer
1.0274
PR #1749by gracebml
1.0996

Hyperparameters Across PRs

pr_numberparameters
1370{"layers":10,"dim":512,"heads":1,"expand_k":1,"expand_v":2}
1371{"layers":9,"gdn_layers":7,"attention_layers":2,"ratio":"3:1","head_dim_ratio":0.75,"expand_v":2,"conv_size":4}
1479{"layers":8}
1553{"layers":10,"head_dim":64,"expand_v":1,"use_short_conv":true}
1562{"layers":5}
1563—
1632{"layers":10}
1749{"layers":10}