← Back to Architecture

DeltaNet

Architecture
Used in
4 PRs
Best BPB
0.7614
Avg BPB
0.8876

Submissions

PR #990by newjordan
0.7614
PR #1028by newjordan
0.8104
PR #1047by newjordan
0.8822
PR #1286by newjordan
1.0963

Hyperparameters Across PRs

pr_numberparameters
990{"heads":4}
1028{"heads":4,"short_conv":true,"loops":4,"flat_layers":4,"crawler_layers":1}
1047{"heads":4}
1286{"layers":8,"final_attention_layer":1,"n_embd":384}