← Back to Architecture

tied embeddings

Architecture
Used in
211 PRs
Best BPB
0.2952
Avg BPB
1.1806

Submissions

PR #30by JackYoung27
1.2663
PR #34by ChenLiu-1996
1.2244
PR #37by khasinski
1.2012
PR #39by nanlliuRECORD
1.2139
PR #42by chonchiog
1.2197
PR #44by daniellawson9999
1.1111
PR #45by kiankyars
1.2240
PR #49by spokane-wayRECORD
1.2058
PR #52by spokane-wayRECORD
1.2014
PR #53by kshitizz36
1.1888
PR #59by notapplica
1.2160
PR #61by saml212
1.2154
PR #63by yahya010RECORD
1.1598
PR #65by aquariouseworkmanRECORD
1.1556
PR #66by arjun-krishna1
1.1632
PR #69by TevBenji
1.1708
PR #70by jfprincz
1.1659
PR #71by AntDX316
1.3509
PR #74by takhir-iota
1.1884
PR #75by takhir-iota
1.1768
PR #79by Marvbuster
1.8698
PR #85by hydeh3r3
1.2156
PR #86by aruniyerRECORD
1.1502
PR #88by seanward
1.1605
PR #93by aamodbhatt
1.3693
PR #95by MatoTeziTanka
1.1836
PR #96by saml212
1.1764
PR #103by MatthewHRockwell
1.5000
PR #107by m0at
1.1648
PR #108by kellyvv
1.4370
PR #110by mr-ashish-panday
1.2244
PR #113by JoeProAI
1.1870
PR #114by saml212
1.1574
PR #123by saikrishnarallabandi
1.1642
PR #125by akshai0296
1.3797
PR #126by Athenox14
1.7510
PR #128by rsavitt
1.1594
PR #131by Billy1900
1.2701
PR #136by ibarrajo
1.2101
PR #139by ksang123
1.2029
PR #142by ankitmaloo
1.1925
PR #143by Julz19
1.1779
PR #145by mrdavtan
1.2052
PR #146by swapp1990
1.2987
PR #147by ankitmaloo
1.1631
PR #150by yahya010
1.1478
PR #151by mrdavtan
1.2045
PR #152by timowhite88
1.1744
PR #156by dexhunter
1.1602
PR #160by ChaseWNorton
1.1623
PR #163by Focus2321
1.2091
PR #166by chinesepowered
1.1550
PR #168by spokane-way
1.0217
PR #169by beee003
1.1973
PR #170by baudrillardsgh0st
1.1669
PR #172by GMaN1911
1.1812
PR #173by tamoghnokandar
1.1532
PR #176by GLDRoger
1.1732
PR #180by thwu1RECORD
1.1428
PR #183by anantdgoel
1.2529
PR #184by Idan3011
1.1855
PR #190by newjordan
1.1725
PR #191by chris-buckley
1.1598
PR #192by baudrillardsgh0st
1.1502
PR #193by KHUCHAN
1.2917
PR #194by baudrillardsgh0st
1.1480
PR #195by chasewebb
1.2355
PR #200by khasinski
1.2012
PR #204by Akasxh
1.2320
PR #206by dexhunter
1.1507
PR #209by JWLBOYCE
1.1624
PR #211by dubthecat
1.1719
PR #215by JayCheng113
1.1548
PR #217by kshitizz36
1.1753
PR #218by bopmite
1.1248
PR #219by alertcat
1.1541
PR #222by ansh-deriv
1.1601
PR #223by 0xjaishy
1.1326
PR #231by lenguyen1807
1.2036
PR #232by kellyvv
1.4370
PR #237by takoyakisoft
1.8389
PR #240by riatzukiza
1.6660
PR #247by riatzukiza
1.6114
PR #248by riatzukiza
1.6231
PR #251by kshitizz36
1.1596
PR #252by greqone
1.1554
PR #256by IvGolovach
1.1779
PR #258by riatzukiza
1.6572
PR #262by ibarrajo
1.0539
PR #263by Dannybc123
1.5382
PR #264by stukenov
1.1455
PR #265by unnir
1.1307
PR #266by User123331
1.3932
PR #267by andrewgcodes
1.1374
PR #273by dentity007
1.1575
PR #274by haikosys
1.1403
PR #275by ibarrajo
1.0539
PR #278by nicolasdickenmann
1.0365
PR #284by DanishjeetSingh
1.4106
PR #294by sseanliu
1.1645
PR #295by gowtham0992
1.1477
PR #296by sseanliu
1.1645
PR #301by lookin-zz
1.1807
PR #306by xuafeng
1.1448
PR #309by NewyorkDev
1.1914
PR #327by Ananddna
1.1450
PR #330by bopmite
1.1609
PR #333by mahsumaktas
1.1565
PR #343by joeynyc
1.2459
PR #344by aryanbhosale
1.1330
PR #349by Mapika
1.1399
PR #351by sp00mm
1.1659
PR #352by sp00mm
1.1659
PR #362by mkenney2
1.1497
PR #367by ksang123
1.1770
PR #368by MatoTeziTanka
1.2037
PR #369by signalrush
1.1328
PR #373by JoeProAI
1.1634
PR #374by unnirRECORD
1.1246
PR #381by codestrongestx
1.1739
PR #383by joelnishanth
1.1320
PR #385by dentity007
1.1488
PR #388by ElliotSlusky
1.1231
PR #390by newjordan
1.1295
PR #393by CrimsonSithria
1.2417
PR #394by greqone
1.1247
PR #401by newjordan
1.1243
PR #406by dentity007
1.1287
PR #407by itu-itis24-buyukhelvacigilm24
1.3208
PR #414by signalrush
1.1233
PR #418by yashverms
1.1715
PR #420by leofeasby
1.1454
PR #426by aniketio-ctrl
1.2026
PR #432by jadechip
1.5295
PR #434by parinzee
1.1370
PR #436by CrimsonSithria
1.2392
PR #437by jupram
1.2257
PR #444by AymanMahfuz27
1.4536
PR #446by sofiabod
1.1933
PR #451by harborglowvintage-oss
1.1464
PR #455by kasimte
1.1299
PR #460by abhishekrajdhar
1.2928
PR #465by LoquiAuris
1.1508
PR #466by simonbissonnette
1.1354
PR #467by ADIITJ
1.1428
PR #470by leofeasby
1.1454
PR #477by harsha-gouru
1.1522
PR #478by gowtham0992
1.1268
PR #481by mrdavtan
1.0970
PR #485by harsha-gouru
1.1522
PR #489by sofiabod
1.1327
PR #492by Divyesh-Thirukonda
1.1591
PR #495by SergiuDeveloper
1.2092
PR #498by newjordan
1.1478
PR #499by newjordan
1.1478
PR #508by newjordan
1.1215
PR #512by MatoTeziTanka
0.9512
PR #516by Asukabot0
1.1428
PR #518by sofiabod
1.0622
PR #525by hypery11
1.1160
PR #532by NotADevIAmaMeatPopsicle
1.0487
PR #550by haimianbaobao007
1.1890
PR #552by loveless2001
1.1634
PR #554by chrisnkuno
1.4612
PR #559by Parswanadh
1.5348
PR #563by instax-dutta
1.1428
PR #568by MatoTeziTanka
0.7853
PR #573by Sarimsaljook
1.0523
PR #587by newjordan
1.1208
PR #588by andyluo22
1.4120
PR #595by LoquiAuris
1.1100
PR #605by bigbag
0.7227
PR #612by Christopher-Lee-McClendon
1.1079
PR #622by Upsalla
1.0941
PR #634by raahilshah
1.1171
PR #636by NewyorkDev
1.1234
PR #649by pall23-mech
1.2073
PR #662by simon-marcus
1.1208
PR #664by tsbiosky
1.2982
PR #665by harborglowvintage-oss
1.1464
PR #666by chrislovescoding
1.1932
PR #669by amabito
1.4942
PR #678by SPThole
1.3525
PR #695by 0xNoramiya
1.1360
PR #705by seanward
1.2151
PR #706by newjordan
1.0461
PR #709by StolbaJ
1.1478
PR #710by Dhruba531
1.1240
PR #716by SHN2004
1.4239
PR #724by hypery11
1.0717
PR #727by Asukabot0
0.9674
PR #730by janwww
1.1570
PR #731by pentxayc
1.0400
PR #754by aryanbhosale
1.1253
PR #758by hypery11
1.0465
PR #763by hypery11
0.9917
PR #769by MatoTeziTanka
0.8508
PR #779by deanbrr
0.6683
PR #793by pall23-mech
1.2500
PR #797by armantsaturian
0.8960
PR #806by ibarrajo
0.6678
PR #808by Naazimsnh02
0.6364
PR #809by AayushBaniya2006
0.2952
PR #810by Idan3011
0.9393
PR #816by jimliu741523
1.1194
PR #822by henrycashe26
1.2604
PR #828by bigbag
0.9076
PR #838by aryanbhosale
1.1215
PR #841by someone114514
1.1157
PR #849by dttdrv
1.1105
PR #858by nickferrantelive
1.2135

Hyperparameters Across PRs

pr_numberparameters
30—
34—
37{"tie_embeddings":0}
39—
42{"tie_embeddings":1}
44—
45—
49—
52—
53{"tie_embeddings":1}
59—
61—
63—
65—
66—
69—
70—
71—
74{"tie_embeddings":1}
75{"tie_embeddings":1}
79—
85—
86{"vocab_size":1024}
88{"fp16_export":1}
93—
95—
96—
103—
107—
108—
110—
113—
114—
123—
125{"enabled":1}
126—
128{"tie_embeddings":1}
131—
136—
139—
142—
143{"tie_embeddings":1}
145{"layers":9,"dimensions":512}
146—
147—
150—
151—
152—
156—
160{"tie_embeddings":1}
163—
166—
168—
169—
170—
172—
173—
176—
180—
183—
184—
190{"fp16":true}
191—
192—
193—
194—
195—
200—
204—
206—
209{"layers":11,"vocab":1024,"dim":512,"heads":8,"kv":4,"mlp_hidden":1536}
211—
215—
217—
218—
219{"vocab":1024}
222{"fp16_passthrough":true}
223—
231—
232—
237—
240—
247—
248—
251—
252{"layers":9,"width":512,"sp":1024}
256—
258—
262—
263{"layers":9,"dim":512,"heads":8,"kv_heads":4,"mlp_multiplier":2}
264—
265—
266—
267—
273—
274—
275—
278—
284—
294{"vocab_size":1024,"dimension":768}
295—
296—
301—
306—
309—
327—
330—
333—
343—
344—
349—
351—
352—
362—
367—
368—
369—
373—
374—
381{"layers":10,"model_dim":512,"num_heads":8,"num_kv_heads":4}
383—
385—
388—
390—
393—
394—
401—
406—
407—
414—
418—
420—
426—
432—
434—
436—
437—
444—
446—
451—
455—
460—
465—
466—
467—
470—
477—
478—
481—
485—
489{"vocab_size":1024}
492—
495{"TIE_EMBEDDINGS":1}
498—
499—
508—
512—
516—
518{"vocab_size":1024}
525—
532—
550—
552—
554—
559{"precision":"FP16"}
563—
568—
573—
587—
588{"TIE_EMBEDDINGS":0}
595—
605—
612—
622—
634—
636—
649—
662—
664—
665—
666—
669—
678—
695—
705—
706—
709{"vocab_size":1024}
710—
716—
724—
727—
730{"embed_dim":254,"vocab_size":8192}
731—
754—
758—
763—
769—
779—
793—
797{"softcap":30}
806—
808—
809—
810—
816—
822—
828—
838—
841—
849—
858—