← Back to Architecture

weight tying

Architecture
Used in
288 PRs
Best BPB
0.0180
Avg BPB
1.1341

Submissions

PR #5by albertorkive
1.2244
PR #41by kiankyars
1.2296
PR #50by mattqlf
1.1925
PR #54by TheCause
1.5283
PR #144by DJLougen
1.3517
PR #197by machdragon
1.1893
PR #212by mrdavtan
1.1329
PR #236by saml212
1.1400
PR #244by simon-marcus
1.2064
PR #272by simon-marcus
1.2427
PR #276by riatzukiza
1.6577
PR #284by DanishjeetSingh
1.4106
PR #294by sseanliu
1.1645
PR #302by JackYoung27
1.1520
PR #305by Naazimsnh02
1.1672
PR #309by NewyorkDev
1.1914
PR #310by vishesh9131
1.1787
PR #319by Arth-Singh
1.2716
PR #321by andreanjos
1.1864
PR #331by Rhodrium
1.1487
PR #355by josusanmartin
1.1929
PR #357by adityagupta26
1.1928
PR #366by shivnarainms22
1.1574
PR #391by NishantDahal
1.2374
PR #395by NishantDahal
1.2374
PR #398by felipe-parodi
1.1213
PR #399by abaybektursun
1.1247
PR #422by albertorkive
1.1396
PR #433by Robby955
1.3441
PR #442by sjp611
1.1027
PR #457by carlesonielfa
1.1839
PR #462by JoeProAI
1.0672
PR #476by aquemy
1.4574
PR #483by tmustier
1.1346
PR #510by SelfAnush
1.1989
PR #532by NotADevIAmaMeatPopsicle
1.0487
PR #536by jaksenc
1.5140
PR #548by LoquiAuris
1.0865
PR #557by hypery11
1.1160
PR #564by sadeghja1070
1.1270
PR #585by EthanYangTW
1.1179
PR #618by 0xtigerclaw
1.4702
PR #633by MatoTeziTanka
1.1526
PR #648by maorinka
1.1428
PR #668by Christopher-Lee-McClendon
1.0920
PR #671by keshav55
1.1807
PR #713by hypery11
1.1180
PR #749by FyeJordy
1.3684
PR #755by dcrow85
1.0321
PR #767by RichiiiTV
0.9209
PR #768by mradassaad
1.1201
PR #773by siddhantparadox
1.1532
PR #783by petergpt
1.1171
PR #785by SirSaltySalmon
1.5364
PR #786by shinegami-2002
0.8128
PR #830by zlxi02
1.4096
PR #832by jfprincz
1.1903
PR #869by THUQiXuan
0.1290
PR #871by greqone
0.8004
PR #875by shalyhinpavel
1.0226
PR #890by sofiabod
0.4405
PR #892by robbiebusinessacc
1.1428
PR #895by iverbovoy
1.0889
PR #903by CiprianFlorin-Ifrim
1.2064
PR #904by anthony-maio
1.2734
PR #905by anthony-maio
1.8587
PR #907by resouer
0.0960
PR #908by albertorkive
1.1734
PR #909by sunnypatneedi
0.8609
PR #913by RoyiRa
0.0887
PR #918by haikosys
0.1653
PR #920by CiprianFlorin-Ifrim
1.1539
PR #921by TimPietrusky
0.0939
PR #923by CiprianFlorin-Ifrim
1.1090
PR #929by andreanjos
1.1653
PR #933by haikosys
0.0804
PR #937by mihir-s-05
1.4457
PR #961by callithyia
0.0881
PR #962by AnirudhRahul
0.0214
PR #963by sunnypatneedi
0.8609
PR #965by Adam-Jacuch
1.1184
PR #967by dexhunter
1.0450
PR #969by dnldsz
1.2907
PR #970by dnldsz
1.2907
PR #972by Idan3011
0.3922
PR #976by Vibes-me
1.2058
PR #978by AnirudhRahul
1.5134
PR #979by 0xadvait
1.1387
PR #981by BurguerJohn
1.4893
PR #986by sofiabod
0.0830
PR #989by alexanderaperry-arch
1.1402
PR #990by newjordan
0.7614
PR #992by TimS-ml
1.4054
PR #993by aerosta
0.9631
PR #994by singhaikshitijjain
1.4315
PR #996by Idan3011
1.1478
PR #997by randy06122001-boop
1.4182
PR #1001by ibarrajo
1.1188
PR #1016by ADIITJ
1.1269
PR #1022by aramdov
1.1646
PR #1030by sofiabod
0.1130
PR #1034by Jeneesh1014
1.7195
PR #1036by ivanontech
1.1974
PR #1038by Vibes-me
1.2058
PR #1040by JoeProAI
1.1336
PR #1041by JoeProAI
1.1356
PR #1044by greqone
1.8989
PR #1046by Jayteare
1.2174
PR #1048by mrdavtan
1.1724
PR #1050by Taleef7
1.1194
PR #1055by sanyalsunny111
0.9693
PR #1056by sofiabod
0.0180
PR #1057by Programmerryoki
1.2201
PR #1059by edidisheng
1.1996
PR #1063by SHN2004
1.3321
PR #1065by rithunkp
1.1536
PR #1074by ldh-at
1.3288
PR #1088by serdardoesml
1.2542
PR #1095by vimeto
0.0905
PR #1096by vimeto
1.3342
PR #1097by danielxmed
1.3355
PR #1106by agalimova
1.1465
PR #1107by mradassaad
1.5633
PR #1110by gowtham0992
1.2249
PR #1123by sisegod
1.1986
PR #1126by AnirudhRahul
1.1091
PR #1142by ymrohit
1.1493
PR #1152by ericdatum
1.7942
PR #1159by JDAppleseed
0.3693
PR #1165by brandonpf
1.2314
PR #1166by Christopher-Lee-McClendon
1.1347
PR #1170by Christopher-Lee-McClendon
1.1199
PR #1180by estesryan
1.0577
PR #1185by skoustav35
0.9641
PR #1193by dentity007
1.4390
PR #1198by ymrohit
1.5992
PR #1212by Gusanidas
1.1108
PR #1221by amabito
1.1915
PR #1226by Wolfie8935
1.1428
PR #1227by himanshudongre
1.4841
PR #1232by Christopher-Lee-McClendon
1.0929
PR #1235by maksblu
1.3527
PR #1236by ibarrajo
1.1179
PR #1241by aiejvn
0.9901
PR #1242by Campbellb
1.0903
PR #1245by mkenney2
1.1470
PR #1254by Elarwei001
1.1070
PR #1258by jorge-asenjo
1.3874
PR #1261by Aniket-pd
1.3029
PR #1263by xexyz
0.9354
PR #1278by GitGeeks
1.1147
PR #1282by newjordan
1.1035
PR #1284by tyrel-beede
1.1207
PR #1286by newjordan
1.0963
PR #1287by dentity007
1.1048
PR #1293by 5en5e1
1.2409
PR #1299by Ribin545
1.8184
PR #1300by Ribin545
1.8184
PR #1307by amrayach
1.1101
PR #1308by newjordan
1.1364
PR #1311by htrung1105
1.1303
PR #1312by adi-suresh01
1.3299
PR #1315by andrewmouldon
1.2207
PR #1322by newjordan
1.0854
PR #1335by WeijieChen2017
1.1948
PR #1337by sergimichi
1.2079
PR #1337by sergimichi
1.2079
PR #1339by bigbag
1.0955
PR #1342by nicholasbailey87
1.4816
PR #1347by shasank0001
1.3038
PR #1355by mradassaad
1.1526
PR #1361by jorge-asenjo
1.1220
PR #1370by Christopher-Lee-McClendon
1.0030
PR #1381by X-Abhishek-X
1.1604
PR #1383by nirmathur
1.3151
PR #1384by iverbovoy
1.1441
PR #1387by Muhammad-Ahmed-Rayyan
1.2919
PR #1388by CiprianFlorin-Ifrim
1.5390
PR #1400by tmancino
1.1035
PR #1403by Rhoahndur
1.3485
PR #1410by izlley
1.1158
PR #1411by Blakethefn
1.5568
PR #1413by dexhunterRECORD
1.0828
PR #1417by BruhTheMomentum
1.3039
PR #1421by X-Abhishek-X
1.0925
PR #1422by swapp1990
1.1172
PR #1431by Idan3011
1.1266
PR #1434by ranausmanai
1.5207
PR #1435by AbhayAnandUCSD
1.0980
PR #1436by DevWizard-Vandan
1.5546
PR #1437by dexhunter
1.0780
PR #1440by Mertyandimata
1.1026
PR #1444by hypnoastic
1.3081
PR #1445by X-Abhishek-X
1.0889
PR #1450by andrewbaggio1
1.0848
PR #1461by viasky657
0.4118
PR #1465by sisegod
1.1381
PR #1471by X-Abhishek-X
1.0866
PR #1472by trhgbao
1.2066
PR #1478by jxgod
1.1995
PR #1479by andrewbaggio1
1.1450
PR #1481by Cayton-Tech
1.3440
PR #1486by AlirezaAlampour
1.6656
PR #1489by joshkmartinez
1.0736
PR #1492by bigbag
1.0810
PR #1493by bigbagRECORD
1.0810
PR #1501by SPThole
1.1159
PR #1502by SPThole
1.1147
PR #1505by Rohan-Abhilash
1.1791
PR #1508by jpfeiffe
1.1135
PR #1512by Itssshikhar
1.1117
PR #1514by dexhunter
1.0798
PR #1515by dexhunter
1.0872
PR #1520by taka6745
1.0824
PR #1527by alphastar1111
1.2026
PR #1533by aryanbhosale
1.0790
PR #1536by dexhunter
1.0775
PR #1539by translatingthename
1.0587
PR #1542by negrurv
1.5363
PR #1543by PavelPaha
1.3286
PR #1544by Abhishek8108
1.0283
PR #1545by Abhishek8108
1.0283
PR #1546by SPThole
1.0850
PR #1547by adityasasidhar
1.1928
PR #1549by dljr-github
1.3220
PR #1550by translatingthename
1.0587
PR #1555by andrewbaggio1
1.0764
PR #1560by dexhunter
1.0741
PR #1562by joshkmartinez
1.0205
PR #1565by Idan3011
1.1036
PR #1568by yuitokyouni
1.1639
PR #1570by yufang67
1.0970
PR #1574by KRGulaj
1.3587
PR #1579by Tonyy1977
1.1372
PR #1583by codemath3000
1.0801
PR #1584by codemath3000
1.0752
PR #1585by codemath3000
1.0639
PR #1585by codemath3000
1.0639
PR #1586by dexhunter
1.0749
PR #1602by SPThole
1.0744
PR #1606by AlirezaAlampour
1.3969
PR #1608by User123331
1.3921
PR #1612by seekerPrice
1.5096
PR #1616by Vickyrrrrrr
1.4100
PR #1617by adityasasidhar
1.2192
PR #1619by AVINASH0052
1.1156
PR #1623by divagr18
1.1942
PR #1624by joshkmartinez
1.0585
PR #1627by mike-ferguson
1.3246
PR #1629by channyzf6
1.0829
PR #1633by joshkmartinez
1.0585
PR #1640by thestbobo
1.1412
PR #1643by mradassaad
1.1473
PR #1650by Jaredcastorena
1.4233
PR #1658by AVINASH0052
1.0810
PR #1661by anderamondarainh-stack
1.1444
PR #1665by mrbese
1.3571
PR #1670by dexhunter
1.0597
PR #1672by andrewbaggio1
1.0119
PR #1676by aazizyan
1.0788
PR #1683by yunoshev
1.1280
PR #1688by Buld1n
1.0809
PR #1689by chris-colinsky
1.0822
PR #1691by AVINASH0052
1.2244
PR #1693by dexhunter
1.0573
PR #1695by X-Abhishek-X
1.0759
PR #1696by kings-crown
1.1224
PR #1699by lsb
1.4831
PR #1704by Buld1n
1.0976
PR #1709by Bananakin1
1.1470
PR #1714by Anakintano
1.0857
PR #1715by G3sparky
1.0809
PR #1720by kiyoaki
1.0818
PR #1722by deborahnelson8788726
0.6580
PR #1723by SlavH
0.5116
PR #1724by Unwindology
1.1803
PR #1725by teslaeco
1.0813
PR #1728by mikeapedia
1.0771
PR #1731by Victory963
1.0785
PR #1735by AjAnubolu
1.0429
PR #1737by sakthivarshans
1.0723
PR #1738by alertcat
1.0354
PR #1748by elad-simbalista
1.2098
PR #1749by gracebml
1.0996
PR #1753by Abhishek-Dalvi410
1.2917
PR #1755by OE-GOD
1.0746
PR #1756by romeerp
1.0651
PR #1759by yijieyuan
1.0799

Hyperparameters Across PRs

pr_numberparameters
5{"physical_layers":null,"logical_layers":null}
41—
50—
54{"recurrent_passes":3}
144—
197—
212—
236—
244—
272—
276—
284—
294—
302—
305{"dim":512}
309—
310—
319{"unique_layers":5,"loops":3}
321—
331—
355—
357—
366—
391—
395—
398—
399—
422—
433{"shared_blocks":3,"virtual_layers":9,"lora_rank":8}
442—
457—
462—
476—
483—
510—
532—
536—
548—
557—
564—
585{"layers":[9,10]}
618—
633—
648{"num_loops":2}
668—
671—
713{"vocab_size":1024}
749—
755—
767—
768{"layers":[5,6,7,8,9,10]}
773{"tie_embeddings":1}
783{"layers":11,"bigram_vocab_size":1536,"cache_layer":7}
785—
786—
830—
832—
869—
871—
875—
890—
892—
895—
903{"vocab_size":8192}
904—
905—
907—
908—
909—
913—
918—
920—
921—
923—
929—
933—
937—
961—
962—
963—
965—
967—
969—
970—
972—
976—
978—
979—
981—
986—
989—
990—
992—
993—
994—
996—
997{"vocab_size":1024}
1001—
1016—
1022—
1030—
1034—
1036—
1038—
1040{"tied":false}
1041{"tie_embeddings":false}
1044—
1046—
1048—
1050—
1055—
1056—
1057—
1059—
1063—
1065—
1074—
1088—
1095—
1096—
1097—
1106—
1107—
1110{"shared_blocks":3}
1123—
1126—
1142—
1152—
1159—
1165—
1166—
1170—
1180—
1185—
1193—
1198—
1212—
1221—
1226—
1227—
1232—
1235—
1236—
1241—
1242—
1245—
1254—
1258—
1261—
1263—
1278—
1282—
1284—
1286—
1287—
1293—
1299—
1300{"steps":1}
1307—
1308{"flat_layers":9,"crawler_layers":1,"crawler_loops":2}
1311—
1312—
1315{"layers":9,"learned_basis_matrices":3,"fixed_random_projections":512}
1322—
1335{"vocab":1024}
1337{"layers":24,"groups":2,"virtual_layers":24,"degree_attn":5,"degree_ffn":2}
1337{"embedding_projection":[1024,128,512]}
1339{"vocab_size":2048}
1342—
1347{"layers":15,"model_dim":512,"num_heads":8,"num_kv_heads":4,"mlp_mult":3}
1355—
1361—
1370—
1381—
1383{"shared_pairs":["3-4","9-10"]}
1384—
1387{"shared_blocks":2}
1388—
1400—
1403—
1410—
1411—
1413—
1417—
1421—
1422—
1431—
1434—
1435—
1436—
1437—
1440—
1444—
1445—
1450—
1461—
1465—
1471—
1472—
1478—
1479—
1481{"vocab_size":1024,"model_dim":512,"bottleneck_ranks":[64,128]}
1486—
1489—
1492—
1493—
1501—
1502—
1505—
1508{"vocab_size":4096}
1512—
1514—
1515—
1520—
1527—
1533—
1536—
1539—
1542—
1543—
1544—
1545—
1546{"embed_dim":416,"model_dim":512}
1547—
1549—
1550—
1555—
1560—
1562—
1565—
1568{"unique_blocks":6,"passes":2,"effective_layers":12,"d_model":672}
1570—
1574—
1579—
1583—
1584—
1585—
1585—
1586—
1602—
1606—
1608—
1612—
1616—
1617—
1619—
1623—
1624—
1627—
1629—
1633—
1640—
1643—
1650—
1658—
1661—
1665{"ngroups":1}
1670—
1672—
1676—
1683—
1688—
1689—
1691—
1693—
1695—
1696—
1699{"vocab_size_plus_mask":1025}
1704—
1709—
1714—
1715—
1720—
1722—
1723—
1724—
1725—
1728—
1731—
1735—
1737—
1738—
1748—
1749—
1753{"tied_embeddings":1}
1755—
1756—
1759—