← Back to Architecture

weight tying

Architecture
Used in
412 PRs
Best BPB
0.0180
Avg BPB
1.1357

Submissions

PR #5by albertorkive
1.2244
PR #41by kiankyars
1.2296
PR #50by mattqlf
1.1925
PR #54by TheCause
1.5283
PR #144by DJLougen
1.3517
PR #197by machdragon
1.1893
PR #212by mrdavtan
1.1329
PR #236by saml212
1.1400
PR #244by simon-marcus
1.2064
PR #272by simon-marcus
1.2427
PR #276by riatzukiza
1.6577
PR #284by DanishjeetSingh
1.4106
PR #294by sseanliu
1.1645
PR #302by JackYoung27
1.1520
PR #305by Naazimsnh02
1.1672
PR #309by NewyorkDev
1.1914
PR #310by vishesh9131
1.1787
PR #319by Arth-Singh
1.2716
PR #321by andreanjos
1.1864
PR #331by Rhodrium
1.1487
PR #355by josusanmartin
1.1929
PR #357by adityagupta26
1.1928
PR #366by shivnarainms22
1.1574
PR #391by NishantDahal
1.2374
PR #395by NishantDahal
1.2374
PR #398by felipe-parodi
1.1213
PR #399by abaybektursun
1.1247
PR #422by albertorkive
1.1396
PR #433by Robby955
1.3441
PR #442by sjp611
1.1027
PR #457by carlesonielfa
1.1839
PR #462by JoeProAI
1.0672
PR #476by aquemy
1.4574
PR #483by tmustier
1.1346
PR #510by SelfAnush
1.1989
PR #532by NotADevIAmaMeatPopsicle
1.0487
PR #536by jaksenc
1.5140
PR #548by LoquiAuris
1.0865
PR #557by hypery11
1.1160
PR #564by sadeghja1070
1.1270
PR #585by EthanYangTW
1.1179
PR #618by 0xtigerclaw
1.4702
PR #633by MatoTeziTanka
1.1526
PR #648by maorinka
1.1428
PR #668by Christopher-Lee-McClendon
1.0920
PR #671by keshav55
1.1807
PR #713by hypery11
1.1180
PR #749by FyeJordy
1.3684
PR #755by dcrow85
1.0321
PR #767by RichiiiTV
0.9209
PR #768by mradassaad
1.1201
PR #773by siddhantparadox
1.1532
PR #783by petergpt
1.1171
PR #785by SirSaltySalmon
1.5364
PR #786by shinegami-2002
0.8128
PR #830by zlxi02
1.4096
PR #832by jfprincz
1.1903
PR #869by THUQiXuan
0.1290
PR #871by greqone
0.8004
PR #875by shalyhinpavel
1.0226
PR #890by sofiabod
0.4405
PR #892by robbiebusinessacc
1.1428
PR #895by iverbovoy
1.0889
PR #903by CiprianFlorin-Ifrim
1.2064
PR #904by anthony-maio
1.2734
PR #905by anthony-maio
1.8587
PR #907by resouer
0.0960
PR #908by albertorkive
1.1734
PR #909by sunnypatneedi
0.8609
PR #913by RoyiRa
0.0887
PR #918by haikosys
0.1653
PR #920by CiprianFlorin-Ifrim
1.1539
PR #921by TimPietrusky
0.0939
PR #923by CiprianFlorin-Ifrim
1.1090
PR #929by andreanjos
1.1653
PR #933by haikosys
0.0804
PR #937by mihir-s-05
1.4457
PR #961by callithyia
0.0881
PR #962by AnirudhRahul
0.0214
PR #963by sunnypatneedi
0.8609
PR #965by Adam-Jacuch
1.1184
PR #967by dexhunter
1.0450
PR #969by dnldsz
1.2907
PR #970by dnldsz
1.2907
PR #972by Idan3011
0.3922
PR #976by Vibes-me
1.2058
PR #978by AnirudhRahul
1.5134
PR #979by 0xadvait
1.1387
PR #981by BurguerJohn
1.4893
PR #986by sofiabod
0.0830
PR #989by alexanderaperry-arch
1.1402
PR #990by newjordan
0.7614
PR #992by TimS-ml
1.4054
PR #993by aerosta
0.9631
PR #994by singhaikshitijjain
1.4315
PR #996by Idan3011
1.1478
PR #997by randy06122001-boop
1.4182
PR #1001by ibarrajo
1.1188
PR #1016by ADIITJ
1.1269
PR #1022by aramdov
1.1646
PR #1030by sofiabod
0.1130
PR #1034by Jeneesh1014
1.7195
PR #1036by ivanontech
1.1974
PR #1038by Vibes-me
1.2058
PR #1040by JoeProAI
1.1336
PR #1041by JoeProAI
1.1356
PR #1044by greqone
1.8989
PR #1046by Jayteare
1.2174
PR #1048by mrdavtan
1.1724
PR #1050by Taleef7
1.1194
PR #1055by sanyalsunny111
0.9693
PR #1056by sofiabod
0.0180
PR #1057by Programmerryoki
1.2201
PR #1059by edidisheng
1.1996
PR #1063by SHN2004
1.3321
PR #1065by rithunkp
1.1536
PR #1074by ldh-at
1.3288
PR #1088by serdardoesml
1.2542
PR #1095by vimeto
0.0905
PR #1096by vimeto
1.3342
PR #1097by danielxmed
1.3355
PR #1106by agalimova
1.1465
PR #1107by mradassaad
1.5633
PR #1110by gowtham0992
1.2249
PR #1123by sisegod
1.1986
PR #1126by AnirudhRahul
1.1091
PR #1142by ymrohit
1.1493
PR #1152by ericdatum
1.7942
PR #1159by JDAppleseed
0.3693
PR #1165by brandonpf
1.2314
PR #1166by Christopher-Lee-McClendon
1.1347
PR #1170by Christopher-Lee-McClendon
1.1199
PR #1180by estesryan
1.0577
PR #1185by skoustav35
0.9641
PR #1193by dentity007
1.4390
PR #1198by ymrohit
1.5992
PR #1212by Gusanidas
1.1108
PR #1221by amabito
1.1915
PR #1226by Wolfie8935
1.1428
PR #1227by himanshudongre
1.4841
PR #1232by Christopher-Lee-McClendon
1.0929
PR #1235by maksblu
1.3527
PR #1236by ibarrajo
1.1179
PR #1241by aiejvn
0.9901
PR #1242by Campbellb
1.0903
PR #1245by mkenney2
1.1470
PR #1254by Elarwei001
1.1070
PR #1258by jorge-asenjo
1.3874
PR #1261by Aniket-pd
1.3029
PR #1263by xexyz
0.9354
PR #1278by GitGeeks
1.1147
PR #1282by newjordan
1.1035
PR #1284by tyrel-beede
1.1207
PR #1286by newjordan
1.0963
PR #1287by dentity007
1.1048
PR #1293by 5en5e1
1.2409
PR #1299by Ribin545
1.8184
PR #1300by Ribin545
1.8184
PR #1307by amrayach
1.1101
PR #1308by newjordan
1.1364
PR #1311by htrung1105
1.1303
PR #1312by adi-suresh01
1.3299
PR #1315by andrewmouldon
1.2207
PR #1322by newjordan
1.0854
PR #1335by WeijieChen2017
1.1948
PR #1337by sergimichi
1.2079
PR #1337by sergimichi
1.2079
PR #1339by bigbag
1.0955
PR #1342by nicholasbailey87
1.4816
PR #1347by shasank0001
1.3038
PR #1355by mradassaad
1.1526
PR #1361by jorge-asenjo
1.1220
PR #1370by Christopher-Lee-McClendon
1.0030
PR #1381by X-Abhishek-X
1.1604
PR #1383by nirmathur
1.3151
PR #1384by iverbovoy
1.1441
PR #1387by Muhammad-Ahmed-Rayyan
1.2919
PR #1388by CiprianFlorin-Ifrim
1.5390
PR #1400by tmancino
1.1035
PR #1403by Rhoahndur
1.3485
PR #1410by izlley
1.1158
PR #1411by Blakethefn
1.5568
PR #1413by dexhunterRECORD
1.0828
PR #1417by BruhTheMomentum
1.3039
PR #1421by X-Abhishek-X
1.0925
PR #1422by swapp1990
1.1172
PR #1431by Idan3011
1.1266
PR #1434by ranausmanai
1.5207
PR #1435by AbhayAnandUCSD
1.0980
PR #1436by DevWizard-Vandan
1.5546
PR #1437by dexhunter
1.0780
PR #1440by Mertyandimata
1.1026
PR #1444by hypnoastic
1.3081
PR #1445by X-Abhishek-X
1.0889
PR #1450by andrewbaggio1
1.0848
PR #1461by viasky657
0.4118
PR #1465by sisegod
1.1381
PR #1471by X-Abhishek-X
1.0866
PR #1472by trhgbao
1.2066
PR #1478by jxgod
1.1995
PR #1479by andrewbaggio1
1.1450
PR #1481by Cayton-Tech
1.3440
PR #1486by AlirezaAlampour
1.6656
PR #1489by joshkmartinez
1.0736
PR #1492by bigbag
1.0810
PR #1493by bigbagRECORD
1.0810
PR #1501by SPThole
1.1159
PR #1502by SPThole
1.1147
PR #1505by Rohan-Abhilash
1.1791
PR #1508by jpfeiffe
1.1135
PR #1512by Itssshikhar
1.1117
PR #1514by dexhunterRECORD
1.0798
PR #1515by dexhunter
1.0872
PR #1520by taka6745
1.0824
PR #1527by alphastar1111
1.2026
PR #1533by aryanbhosale
1.0790
PR #1536by dexhunter
1.0775
PR #1539by translatingthename
1.0587
PR #1542by negrurv
1.5363
PR #1543by PavelPaha
1.3286
PR #1544by Abhishek8108
1.0283
PR #1545by Abhishek8108
1.0283
PR #1546by SPThole
1.0850
PR #1547by adityasasidhar
1.1928
PR #1549by dljr-github
1.3220
PR #1550by translatingthename
1.0587
PR #1555by andrewbaggio1
1.0764
PR #1560by dexhunter
1.0741
PR #1562by joshkmartinez
1.0205
PR #1565by Idan3011
1.1036
PR #1568by yuitokyouni
1.1639
PR #1570by yufang67
1.0970
PR #1574by KRGulaj
1.3587
PR #1579by Tonyy1977
1.1372
PR #1583by codemath3000
1.0801
PR #1584by codemath3000
1.0752
PR #1585by codemath3000
1.0639
PR #1585by codemath3000
1.0639
PR #1586by dexhunter
1.0749
PR #1602by SPThole
1.0744
PR #1606by AlirezaAlampour
1.3969
PR #1608by User123331
1.3921
PR #1612by seekerPrice
1.5096
PR #1616by Vickyrrrrrr
1.4100
PR #1617by adityasasidhar
1.2192
PR #1619by AVINASH0052
1.1156
PR #1623by divagr18
1.1942
PR #1624by joshkmartinez
1.0585
PR #1627by mike-ferguson
1.3246
PR #1629by channyzf6
1.0829
PR #1633by joshkmartinez
1.0585
PR #1640by thestbobo
1.1412
PR #1643by mradassaad
1.1473
PR #1650by Jaredcastorena
1.4233
PR #1658by AVINASH0052
1.0810
PR #1661by anderamondarainh-stack
1.1444
PR #1665by mrbese
1.3571
PR #1670by dexhunter
1.0597
PR #1672by andrewbaggio1
1.0119
PR #1676by aazizyan
1.0788
PR #1683by yunoshev
1.1280
PR #1688by Buld1n
1.0809
PR #1689by chris-colinsky
1.0822
PR #1691by AVINASH0052
1.2244
PR #1693by dexhunter
1.0573
PR #1695by X-Abhishek-X
1.0759
PR #1696by kings-crown
1.1224
PR #1699by lsb
1.4831
PR #1704by Buld1n
1.0976
PR #1709by Bananakin1
1.1470
PR #1714by Anakintano
1.0857
PR #1715by G3sparky
1.0809
PR #1720by kiyoaki
1.0818
PR #1722by deborahnelson8788726
0.6580
PR #1723by SlavH
0.5116
PR #1724by Unwindology
1.1803
PR #1725by teslaeco
1.0813
PR #1728by mikeapedia
1.0771
PR #1731by Victory963
1.0785
PR #1735by AjAnubolu
1.0429
PR #1737by sakthivarshans
1.0723
PR #1738by alertcat
1.0354
PR #1748by elad-simbalista
1.2098
PR #1749by gracebml
1.0996
PR #1753by Abhishek-Dalvi410
1.2917
PR #1755by OE-GOD
1.0746
PR #1756by romeerp
1.0651
PR #1759by yijieyuan
1.0799
PR #1762by frido22
1.5200
PR #1768by lkk688
1.2792
PR #1769by dexhunterRECORD
1.0645
PR #1770by liujshi
1.0796
PR #1771by bigbag
1.0651
PR #1773by Amanbig
1.1872
PR #1776by anmarhindi
1.0808
PR #1780by wisebreadloaf
1.0806
PR #1782by OE-GOD
1.4131
PR #1783by ismailntl
1.1716
PR #1792by renqianluo
1.0701
PR #1794by Programmerryoki
1.0849
PR #1798by leon2k2k2k
1.0629
PR #1799by jamesEmerson112
1.2073
PR #1802by aamodbhatt
1.0771
PR #1809by PranavViswanath
1.0800
PR #1811by peytontolbert
1.2350
PR #1812by EthanNing
1.0729
PR #1813by djeidy
0.9417
PR #1816by JiaJunDeng5930
1.3915
PR #1818by taka6745
1.1009
PR #1822by Unwindology
1.1785
PR #1828by 5en5e1
1.1169
PR #1834by ghrua
1.0803
PR #1839by cardona
1.1920
PR #1842by Azura-Whiston
1.8838
PR #1845by AlirezaAlampour
1.3095
PR #1850by someone114514
1.0050
PR #1851by aquariouseworkmanRECORD
1.0613
PR #1852by G3sparky
1.0282
PR #1854by ndokutovich
0.9024
PR #1858by G3sparky
0.9946
PR #1862by Hetul803
1.1877
PR #1866by deborahnelson8788726
1.5042
PR #1868by Christopher-Lee-McClendon
1.0615
PR #1876by Meirzhan05
1.0801
PR #1882by jmattew
1.2013
PR #1886by renqianluo
1.0696
PR #1891by peytontolbert
1.2205
PR #1895by VFYAS
1.0785
PR #1897by Blitzo125
1.1636
PR #1898by X-Abhishek-X
1.0661
PR #1913by Jeffrey-Le
1.0847
PR #1914by Fija
1.0612
PR #1917by Blitzo125
1.1636
PR #1919by dev-pratap-singh
1.0587
PR #1922by divagr18
1.1875
PR #1923by jorge-asenjo
1.0658
PR #1932by PrzemyslaV88
1.0796
PR #1933by deborahnelson8788726
0.9915
PR #1934by liujshi
1.0599
PR #1936by hilbertmeng
1.0769
PR #1937by estesryan
1.2058
PR #1938by lijuncheng16
1.0713
PR #1940by skamalad
1.3066
PR #1942by dcynth
1.2257
PR #1944by dmitriymyan1
1.2551
PR #1944by dmitriymyan1
1.2551
PR #1947by phfarath
1.0890
PR #1948by TimS-ml
1.0624
PR #1949by ymrohit
1.1908
PR #1951by DesperateTomatoCultivator
1.3522
PR #1953by andrewbaggio1RECORD
1.0586
PR #1956by AayushBaniya2006
1.0604
PR #1957by mhlov000111
1.2313
PR #1958by okezue
1.0135
PR #1959by remg1997
0.9962
PR #1965by himanshudongre
1.0587
PR #1967by ndokutovich
1.0585
PR #1971by BharathSShankar
1.0750
PR #1972by BharathSShankar
1.0398
PR #1973by harborglowvintage-oss
1.2193
PR #1975by RishabhPrakash5
1.2111
PR #1977by sahiee-dev
1.0730
PR #1978by EthanYangTW
1.0778
PR #1983by harborglowvintage-oss
1.1586
PR #1984by unowenmaxwen
1.0602
PR #1985by yigengjiang
1.1093
PR #1987by TimS-ml
1.0618
PR #1991by joshuaswanson
0.9429
PR #1992by jamesEmerson112
1.0511
PR #1994by potatonyliu
1.3004
PR #1995by User123331
1.0878
PR #2000by edidiongumoh
1.3302
PR #2004by corbensorenson
1.3569
PR #2005by jamesEmerson112
1.0805
PR #2006by Elubrazione
1.0590
PR #2007by Elubrazione
1.0590
PR #2009by SlavH
1.0500
PR #2011by BharathSShankar
1.0750
PR #2015by Muhtasham
1.2880
PR #2017by Armigerous
1.0611
PR #2017by Armigerous
1.0611
PR #2020by Itssshikhar
1.0596
PR #2022by BharathSShankar
1.0720
PR #2026by RahimMirani
1.0611
PR #2027by H1cSuNtDr4C0n3S
1.0806
PR #2028by Arnie016
1.0898
PR #2033by ayushozha
1.0587
PR #2037by organic-intelligence-1976
1.2670
PR #2040by FF-GardenFn
1.2543
PR #2041by jorge-asenjo
1.0569
PR #2042by FF-GardenFn
1.3641
PR #2044by FF-GardenFn
1.3978
PR #2047by ZanePeycke
1.0591
PR #2048by kineticforge
1.3551
PR #2050by AidenGeunGeun
1.0608
PR #2055by twistloop
1.1400
PR #2058by pranavxiyer
1.1971
PR #2060by S0urC10ud
1.0579
PR #2062by BumaldaOverTheWater94
1.2195
PR #2064by Gary0302
1.3779
PR #2067by jiashenggu
1.0592
PR #2071by jamesEmerson112
1.0066
PR #2072by wfproc
1.0717
PR #2077by tranphat180603
1.0877
PR #2078by hi-aduek
1.0580
PR #2080by NewyorkDev
0.9727
PR #2081by maxrubin629
1.1887
PR #2085by umshahid
1.0857
PR #2086by deniskurlov
1.1384
PR #2088by MaxIv25
1.0744
PR #2089by AlirezaAlampour
1.2093
PR #2090by SPThole
1.2310

Hyperparameters Across PRs

pr_numberparameters
5{"physical_layers":null,"logical_layers":null}
41—
50—
54{"recurrent_passes":3}
144—
197—
212—
236—
244—
272—
276—
284—
294—
302—
305{"dim":512}
309—
310—
319{"unique_layers":5,"loops":3}
321—
331—
355—
357—
366—
391—
395—
398—
399—
422—
433{"shared_blocks":3,"virtual_layers":9,"lora_rank":8}
442—
457—
462—
476—
483—
510—
532—
536—
548—
557—
564—
585{"layers":[9,10]}
618—
633—
648{"num_loops":2}
668—
671—
713{"vocab_size":1024}
749—
755—
767—
768{"layers":[5,6,7,8,9,10]}
773{"tie_embeddings":1}
783{"layers":11,"bigram_vocab_size":1536,"cache_layer":7}
785—
786—
830—
832—
869—
871—
875—
890—
892—
895—
903{"vocab_size":8192}
904—
905—
907—
908—
909—
913—
918—
920—
921—
923—
929—
933—
937—
961—
962—
963—
965—
967—
969—
970—
972—
976—
978—
979—
981—
986—
989—
990—
992—
993—
994—
996—
997{"vocab_size":1024}
1001—
1016—
1022—
1030—
1034—
1036—
1038—
1040{"tied":false}
1041{"tie_embeddings":false}
1044—
1046—
1048—
1050—
1055—
1056—
1057—
1059—
1063—
1065—
1074—
1088—
1095—
1096—
1097—
1106—
1107—
1110{"shared_blocks":3}
1123—
1126—
1142—
1152—
1159—
1165—
1166—
1170—
1180—
1185—
1193—
1198—
1212—
1221—
1226—
1227—
1232—
1235—
1236—
1241—
1242—
1245—
1254—
1258—
1261—
1263—
1278—
1282—
1284—
1286—
1287—
1293—
1299—
1300{"steps":1}
1307—
1308{"flat_layers":9,"crawler_layers":1,"crawler_loops":2}
1311—
1312—
1315{"layers":9,"learned_basis_matrices":3,"fixed_random_projections":512}
1322—
1335{"vocab":1024}
1337{"layers":24,"groups":2,"virtual_layers":24,"degree_attn":5,"degree_ffn":2}
1337{"embedding_projection":[1024,128,512]}
1339{"vocab_size":2048}
1342—
1347{"layers":15,"model_dim":512,"num_heads":8,"num_kv_heads":4,"mlp_mult":3}
1355—
1361—
1370—
1381—
1383{"shared_pairs":["3-4","9-10"]}
1384—
1387{"shared_blocks":2}
1388—
1400—
1403—
1410—
1411—
1413—
1417—
1421—
1422—
1431—
1434—
1435—
1436—
1437—
1440—
1444—
1445—
1450—
1461—
1465—
1471—
1472—
1478—
1479—
1481{"vocab_size":1024,"model_dim":512,"bottleneck_ranks":[64,128]}
1486—
1489—
1492—
1493—
1501—
1502—
1505—
1508{"vocab_size":4096}
1512—
1514—
1515—
1520—
1527—
1533—
1536—
1539—
1542—
1543—
1544—
1545—
1546{"embed_dim":416,"model_dim":512}
1547—
1549—
1550—
1555—
1560—
1562—
1565—
1568{"unique_blocks":6,"passes":2,"effective_layers":12,"d_model":672}
1570—
1574—
1579—
1583—
1584—
1585—
1585—
1586—
1602—
1606—
1608—
1612—
1616—
1617—
1619—
1623—
1624—
1627—
1629—
1633—
1640—
1643—
1650—
1658—
1661—
1665{"ngroups":1}
1670—
1672—
1676—
1683—
1688—
1689—
1691—
1693—
1695—
1696—
1699{"vocab_size_plus_mask":1025}
1704—
1709—
1714—
1715—
1720—
1722—
1723—
1724—
1725—
1728—
1731—
1735—
1737—
1738—
1748—
1749—
1753{"tied_embeddings":1}
1755—
1756—
1759—
1762—
1768{"layers":3}
1769—
1770—
1771—
1773—
1776—
1780—
1782—
1783—
1792—
1794—
1798—
1799—
1802—
1809—
1811—
1812—
1813—
1816—
1818—
1822—
1828—
1834{"n_gram":3,"layers":2,"d_hidden":64,"d_embed":64}
1839—
1842—
1845—
1850—
1851—
1852—
1854—
1858—
1862—
1866{"vocab_size":1024}
1868—
1876—
1882—
1886—
1891—
1895—
1897—
1898—
1913—
1914—
1917—
1919—
1922—
1923—
1932—
1933—
1934—
1936—
1937—
1938—
1940—
1942—
1944{"layers":[4,5]}
1944—
1947—
1948{"vocab_size":8192}
1949—
1951—
1953—
1956—
1957—
1958—
1959—
1965—
1967—
1971—
1972—
1973—
1975—
1977—
1978—
1983—
1984—
1985—
1987—
1991—
1992—
1994—
1995—
2000—
2004{"model_dim":704,"factored_embed_dim":832}
2005—
2006—
2007—
2009—
2011—
2015—
2017{"d_seed":64,"d_hidden":256,"ctx_window":3}
2017—
2020—
2022—
2026—
2027—
2028—
2033—
2037—
2040—
2041—
2042—
2044—
2047—
2048—
2050—
2055—
2058—
2060—
2062—
2064—
2067—
2071—
2072{"layers":10}
2077—
2078—
2080—
2081{"loops":3,"shared_blocks":2}
2085—
2086{"embed_bottleneck":"380->576","vocab_size":16384}
2088—
2089—
2090{"shared_blocks":2}