← Back to Architecture

LeakyReLU

Architecture
Used in
294 PRs
Best BPB
0.0180
Avg BPB
1.0373

Submissions

PR #344by aryanbhosale
1.1330
PR #493by parinzee
1.1309
PR #659by deanbrr
1.0920
PR #662by simon-marcus
1.1208
PR #685by andrewbaggio1
1.0366
PR #715by Asukabot0
1.0337
PR #722by magicjulio
0.5588
PR #727by Asukabot0
0.9674
PR #731by pentxayc
1.0400
PR #762by robinojw
0.7139
PR #779by deanbrr
0.6683
PR #825by hypery11
0.5440
PR #840by quietsmile
0.2873
PR #862by grim-hitman0XX
1.3036
PR #864by aryanbhosale
0.2841
PR #865by aryanbhosale
0.2841
PR #870by simon-marcus
0.0935
PR #871by greqone
0.8004
PR #872by gowtham0992
1.0467
PR #876by Bortlesboat
0.5863
PR #880by RoyiRa
0.1003
PR #882by IshiPareek
1.3762
PR #884by BhatiaUday
1.1448
PR #885by lolrazh
0.9958
PR #887by anthony-maio
0.9642
PR #889by anthony-maio
0.9642
PR #890by sofiabod
0.4405
PR #891by robbiebusinessacc
1.1428
PR #892by robbiebusinessacc
1.1428
PR #893by aryanbhosale
0.1310
PR #896by MVPandey
1.1896
PR #900by Robby955
0.1156
PR #907by resouer
0.0960
PR #909by sunnypatneedi
0.8609
PR #912by Bortlesboat
0.3461
PR #914by mkenney2
1.1873
PR #915by anthony-maio
0.9642
PR #916by Bortlesboat
0.3461
PR #918by haikosys
0.1653
PR #921by TimPietrusky
0.0939
PR #924by THUQiXuan
0.0280
PR #925by THUQiXuan
0.0281
PR #926by NandhuRajRK
0.8705
PR #929by andreanjos
1.1653
PR #932by anthony-maio
1.1580
PR #933by haikosys
0.0804
PR #940by antaloaalonso
0.9581
PR #941by aptsalt
1.3620
PR #945by TimPietrusky
0.0274
PR #948by dentity007
0.1156
PR #953by dexhunter
1.0722
PR #961by callithyia
0.0881
PR #964by vivekvar-dl
1.3900
PR #967by dexhunter
1.0450
PR #968by dentity007
0.1154
PR #969by dnldsz
1.2907
PR #970by dnldsz
1.2907
PR #974by anthony-maio
1.6542
PR #975by Abhishek8108
1.1216
PR #977by michaelwinczuk
1.1185
PR #979by 0xadvait
1.1387
PR #986by sofiabod
0.0830
PR #995by dexhunter
1.0362
PR #999by aamodbhatt
1.1179
PR #1001by ibarrajo
1.1188
PR #1002by SoHarshh
1.1650
PR #1005by OnlyJundong
1.0853
PR #1006by NewyorkDev
1.1085
PR #1007by dillon-blake
1.2252
PR #1008by monkeyKingProgrammer
1.1538
PR #1009by SoHarshh
1.1574
PR #1013by himanshudongre
1.1682
PR #1016by ADIITJ
1.1269
PR #1019by abaybektursunRECORD
1.1147
PR #1026by danielxmed
1.0945
PR #1027by Syed-M-Zeeshan
1.3036
PR #1031by michaelwinczuk
1.1185
PR #1032by wfproc
1.3631
PR #1033by Naazimsnh02
0.4311
PR #1037by TimPietruskyRunPod
1.1179
PR #1039by yufengli-oai
1.1184
PR #1043by okezue
1.1261
PR #1044by greqone
1.8989
PR #1052by demouo
1.1978
PR #1056by sofiabod
0.0180
PR #1057by Programmerryoki
1.2201
PR #1060by dexhunter
1.1123
PR #1062by yaowubarbara
1.4508
PR #1063by SHN2004
1.3321
PR #1069by manfromnowhere143
1.1190
PR #1070by manfromnowhere143
1.1190
PR #1072by vimeto
1.1170
PR #1077by malc3om
1.1130
PR #1081by michaelwinczuk
1.1220
PR #1084by AnubhavBharadwaaj
1.1185
PR #1086by Omrigotlieb
1.1349
PR #1087by Dhenenjay
1.1407
PR #1089by mikeapedia
1.1086
PR #1092by teddyoweh
1.1219
PR #1094by michaelwinczuk
0.4027
PR #1095by vimeto
0.0905
PR #1096by vimeto
1.3342
PR #1097by danielxmed
1.3355
PR #1099by Bortlesboat
1.1133
PR #1105by abaybektursun
1.2208
PR #1107by mradassaad
1.5633
PR #1108by DbBested
1.1502
PR #1110by gowtham0992
1.2249
PR #1111by MichaelMcCulloch
0.2532
PR #1112by dillon-blake
1.2252
PR #1113by gowtham0992
1.3705
PR #1114by minh-stakc
0.0235
PR #1119by gowtham0992
1.4584
PR #1122by icryo
1.1146
PR #1123by sisegod
1.1986
PR #1125by jainpranjal97
1.1946
PR #1126by AnirudhRahul
1.1091
PR #1128by AnubhavBharadwaaj
1.1154
PR #1130by Gusanidas
1.1140
PR #1131by JianYan11
1.2947
PR #1135by barneywohl
1.1116
PR #1136by Hieuabssy
1.3069
PR #1137by Hieuabssy
1.3069
PR #1148by aamodbhatt
1.1179
PR #1150by sahiee-dev
1.1151
PR #1166by Christopher-Lee-McClendon
1.1347
PR #1169by Bortlesboat
1.1126
PR #1170by Christopher-Lee-McClendon
1.1199
PR #1171by EthanYangTW
1.1145
PR #1172by dexhunter
1.1015
PR #1176by bigbag
1.0962
PR #1182by adityakm24
1.1227
PR #1183by akaiHuang
1.5080
PR #1184by icryo
0.9485
PR #1185by skoustav35
0.9641
PR #1186by andrewbaggio1
0.9850
PR #1192by dentity007
1.3560
PR #1202by VirajDeshwal
1.1412
PR #1209by andrewbaggio1
1.1064
PR #1212by Gusanidas
1.1108
PR #1215by turbo-indubitable
1.1601
PR #1221by amabito
1.1915
PR #1222by abaybektursun
1.4707
PR #1230by nestamidavaine
1.1163
PR #1231by nestamidavaine
1.1163
PR #1232by Christopher-Lee-McClendon
1.0929
PR #1239by tmancino
1.5918
PR #1240by andrewbaggio1
1.1064
PR #1243by simon-marcus
1.1230
PR #1244by monkeyKingProgrammer
1.1443
PR #1245by mkenney2
1.1470
PR #1252by ahmetdenizyilmaz
1.0713
PR #1253by Okropniak
1.2326
PR #1254by Elarwei001
1.1070
PR #1255by akaiHuang
1.5080
PR #1256by oidebrett
1.1444
PR #1259by himanshudongre
1.1533
PR #1263by xexyz
0.9354
PR #1269by Jtss-ux
1.1194
PR #1274by MatoTeziTanka
1.0876
PR #1284by tyrel-beede
1.1207
PR #1289by MatoTeziTanka
1.0819
PR #1290by aryanbhosale
1.1104
PR #1296by aryanbhosale
1.0926
PR #1299by Ribin545
1.8184
PR #1302by vlivashkin
1.1078
PR #1303by anthony-maio
0.9462
PR #1307by amrayach
1.1101
PR #1308by newjordan
1.1364
PR #1310by cadenmcmann
1.1177
PR #1311by htrung1105
1.1303
PR #1312by adi-suresh01
1.3299
PR #1313by anthony-maio
0.8637
PR #1318by renqianluo
1.0095
PR #1319by canivel
0.6951
PR #1320by jpfeiffe
1.1196
PR #1321by anthony-maio
0.7406
PR #1322by newjordan
1.0854
PR #1324by yahya010
0.8275
PR #1325by monisha-max
1.3868
PR #1328by renqianluo
0.6361
PR #1329by renqianluo
0.6361
PR #1330by luciobaiocchi
1.4617
PR #1355by mradassaad
1.1526
PR #1359by LucasErcolano
0.4188
PR #1364by stukenov
1.1025
PR #1368by JKSNS
0.8503
PR #1376by stukenov
0.7094
PR #1378by Rajat123456789
1.1711
PR #1379by LucasErcolano
0.4162
PR #1383by nirmathur
1.3151
PR #1384by iverbovoy
1.1441
PR #1386by Buld1n
1.1452
PR #1389by Rome-1
1.7270
PR #1390by NICOH-YAY
1.2634
PR #1395by dttdrv
1.0924
PR #1399by AnubhavBharadwaaj
1.0898
PR #1400by tmancino
1.1035
PR #1405by anthony-maio
1.0856
PR #1406by aamodbhatt
1.0887
PR #1407by OnlyJundong
1.0960
PR #1410by izlley
1.1158
PR #1413by dexhunterRECORD
1.0828
PR #1414by Abhishek8108
0.7093
PR #1420by abaybektursun
1.0801
PR #1424by OnlyJundong
1.0858
PR #1427by kjahan
1.2092
PR #1431by Idan3011
1.1266
PR #1437by dexhunter
1.0780
PR #1438by sabdulmajid
1.2029
PR #1440by Mertyandimata
1.1026
PR #1442by akaiHuang
1.1854
PR #1444by hypnoastic
1.3081
PR #1446by LauraGomezjurado
1.0960
PR #1450by andrewbaggio1
1.0848
PR #1453by iverbovoy
1.1324
PR #1456by sisegod
1.1465
PR #1457by DilpreetBansi
1.1454
PR #1471by X-Abhishek-X
1.0866
PR #1472by trhgbao
1.2066
PR #1473by AVINASH0052
1.1156
PR #1484by AlirezaAlampour
1.6656
PR #1485by ndokutovich
1.0679
PR #1486by AlirezaAlampour
1.6656
PR #1492by bigbag
1.0810
PR #1493by bigbagRECORD
1.0810
PR #1494by G3sparky
1.1220
PR #1499by dippatel1994
1.6323
PR #1507by ChideraIbe123
0.2282
PR #1514by dexhunter
1.0798
PR #1515by dexhunter
1.0872
PR #1517by RulinShao
1.0632
PR #1520by taka6745
1.0824
PR #1523by EthanYangTW
1.0778
PR #1527by alphastar1111
1.2026
PR #1528by xiehuanyi
1.1104
PR #1530by samacqua
1.0764
PR #1537by pireylow
1.3971
PR #1538by davie2009kh
1.1180
PR #1539by translatingthename
1.0587
PR #1540by aryanbhosale
1.0777
PR #1541by bigbag
1.0778
PR #1546by SPThole
1.0850
PR #1548by dljr-github
1.3220
PR #1549by dljr-github
1.3220
PR #1550by translatingthename
1.0587
PR #1555by andrewbaggio1
1.0764
PR #1560by dexhunter
1.0741
PR #1561by EthanYangTW
1.0783
PR #1565by Idan3011
1.1036
PR #1568by yuitokyouni
1.1639
PR #1573by shivangbaveja
1.1464
PR #1581by aiejvn
1.2321
PR #1583by codemath3000
1.0801
PR #1584by codemath3000
1.0752
PR #1585by codemath3000
1.0639
PR #1586by dexhunter
1.0749
PR #1606by AlirezaAlampour
1.3969
PR #1616by Vickyrrrrrr
1.4100
PR #1619by AVINASH0052
1.1156
PR #1621by mrbese
1.1531
PR #1626by dexhunter
1.0719
PR #1628by yu314-coder
1.1921
PR #1629by channyzf6
1.0829
PR #1639by kunwar-vikrant
1.0832
PR #1640by thestbobo
1.1412
PR #1643by mradassaad
1.1473
PR #1646by sergeevii123
1.0909
PR #1647by powerpratik
1.0616
PR #1648by mikeapedia
1.0756
PR #1650by Jaredcastorena
1.4233
PR #1655by himanalot
1.1135
PR #1660by pablinga19
1.0858
PR #1666by mrbese
1.1531
PR #1667by MarioPaerle
1.0714
PR #1670by dexhunter
1.0597
PR #1675by jayzuccarelli
1.1451
PR #1676by aazizyan
1.0788
PR #1679by ChideraIbe123
0.7625
PR #1688by Buld1n
1.0809
PR #1689by chris-colinsky
1.0822
PR #1693by dexhunter
1.0573
PR #1696by kings-crown
1.1224
PR #1709by Bananakin1
1.1470
PR #1714by Anakintano
1.0857
PR #1715by G3sparky
1.0809
PR #1716by himanshudongre
1.0788
PR #1720by kiyoaki
1.0818
PR #1731by Victory963
1.0785
PR #1737by sakthivarshans
1.0723
PR #1754by upascal
1.0881
PR #1755by OE-GOD
1.0746
PR #1759by yijieyuan
1.0799
PR #1760by BrandtChristian
1.1863

Hyperparameters Across PRs

pr_numberparameters
344{"slope":0.5,"power":2}
493{"negative_slope":0.5,"power":2}
659{"slope":0.5,"power":2}
662{"negative_slope":0.5,"power":2}
685{"negative_slope":0.5}
715{"negative_slope":0.5,"squared":true}
722{"negative_slope":0.5}
727{"negative_slope":0.5}
731{"negative_slope":0.5,"power":2}
762{"slope":0.5}
779{"slope":0.5}
825{"negative_slope":0.5,"power":2}
840{"slope":0.9,"power":2}
862{"negative_slope":0.5}
864{"multiplier":3,"slope":0.5}
865{"multiplier":3,"slope":0.5}
870{"negative_slope":0.5}
871{"multiplier":"3x","squared":true,"slope":0.5}
872{"layers":3}
876{"slope":0.5}
880{"slope":0.5}
882{"negative_slope":0.5}
884{"negative_slope":0.5}
885{"negative_slope":0.9}
887{"power":2,"slope":0.5}
889{"power":2,"slope":0.5}
890{"slope":0.5}
891{"negative_slope":0.5}
892{"slope":0.5}
893{"mlp_multiplier":"3x","power":2,"slope":0.5}
896{"slope":0.5}
900—
907{"squared":true,"slope":0.9}
909{"slope":0.5}
912{"slope":0.5}
914{"slope":0.9}
915{"power":2}
916{"slope":0.5}
918{"squared":true,"negative_slope":0.5}
921{"negative_slope":0.5}
924{"squared":true}
925{"squared":true}
926{"slope":0.5,"squared":true}
929{"negative_slope":0.5}
932{"slope":0.5}
933{"negative_slope":0.5}
940{"negative_slope":0.5}
941{"squared":true,"negative_slope":0.5}
945{"squared":true,"negative_slope":0.5}
948{"negative_slope":0.5}
953{"squared":true,"alpha":0.5}
961{"negative_slope":0.5,"squared":true}
964{"squared":true,"negative_slope":0.5}
967{"squared":true,"negative_slope":0.5}
968{"slope":0.5}
969{"slope":0.5}
970{"slope":0.5}
974{"squared":true,"negative_slope":0.5}
975{"slope":0.5}
977{"negative_slope":0.75}
979{"squared":true,"negative_slope":0.5}
986{"squared":true,"slope":0.5}
995{"slope":0.5}
999{"slope":0.5}
1001—
1002{"slope":0.5}
1005{"squared":true,"slope":0.5}
1006{"squared":true,"negative_slope":0.5}
1007{"slope":0.5}
1008{"slope":0.5,"power":2}
1009{"squared":true,"negative_slope":0.5}
1013—
1016{"slope":0.5}
1019{"squared":true}
1026{"squared":true,"negative_slope":0.5}
1027{"negative_slope":0.5}
1031{"negative_slope":0.75}
1032{"slope":0.75}
1033{"slope":0.5}
1037{"negative_slope":0.5}
1039{"power":2,"slope":0.5}
1043{"power":2}
1044{"slope":0.5}
1052{"power":2,"slope":0.5}
1056{"squared":true,"negative_slope":0.5}
1057{"mlp_mult":2,"negative_slope":0.5,"squared":true}
1060{"layers":11}
1062{"negative_slope":0.9}
1063{"layers":9,"width":512,"heads":8,"kv_heads":4,"mlp_mult":2}
1069{"slope":0.5}
1070{"squared":true,"negative_slope":0.5}
1072{"slope":0.5,"squared":true}
1077{"slope":0.5}
1081{"squared":true,"slope":0.75}
1084{"squared":true}
1086{"slope":0.5}
1087{"slope":0.5,"squared":true}
1089{"slope":0.3,"squared":true}
1092{"negative_slope":0.75}
1094{"power":2,"slope":0.75}
1095{"slope":0.5,"squared":true}
1096{"slope":0.5,"squared":true}
1097{"negative_slope":0.5}
1099—
1105{"negative_slope":0.5}
1107—
1108{"squared":true,"negative_slope":0.5}
1110{"slope":0.5}
1111—
1112{"slope":0.5}
1113{"slope":0.5}
1114{"mlp_multiplier":3,"activation_power":2,"slope":0.5}
1119{"negative_slope":0.5,"squared":true}
1122{"negative_slope":0.3,"squared":true}
1123{"negative_slope":0.5}
1125{"alpha":0.5,"squared":true}
1126{"negative_slope":0.3}
1128{"mlp_layers":3}
1130{"squared":true,"negative_slope":0.5}
1131{"negative_slope":0.5}
1135{"activation":"leaky_relu(x, 0.5).square()"}
1136{"slope":0.5}
1137{"slope":0.5,"squared":true}
1148{"slope":0.5}
1150{"squared":true,"mlp_mult":2.8}
1166{"slope":0.5}
1169{"negative_slope":0.3}
1170{"variant":"squared","slope":0.5}
1171—
1172{"slope":0.5}
1176{"slope":0.5}
1182{"squared":true,"negative_slope":0.5}
1183{"negative_slope":0.5}
1184—
1185{"squared":true,"negative_slope":0.5}
1186{"slope":0.5}
1192{"slope":0.75}
1202{"squared":true,"alpha":0.5}
1209{"negative_slope":0.5}
1212{"slope":0.5}
1215{"slope":0.95,"power":2}
1221{"negative_slope":0.5}
1222{"layers":3,"rank":256,"prime_layers":[8,9,10]}
1230{"slope":0.5}
1231{"variant":"LeakyReLU(0.5)^2"}
1232{"slope":0.5}
1239{"slope":0.5}
1240{"slope":0.5}
1243{"slope":0.5}
1244{"layers":11,"d_model":512,"heads":8,"kv_heads":4}
1245{"slope":0.9}
1252{"squared":true,"negative_slope":0.5}
1253{"slope":0.1}
1254{"slope":0.5}
1255{"slope":0.5}
1256{"negative_slope":0.5}
1259—
1263{"negative_slope":0.5,"squared":true,"mlp_expansion":3}
1269{"negative_slope":0.5,"power":2}
1274{"negative_slope":0.5}
1284{"slope":0.5}
1289{"squared":true,"negative_slope":0.5}
1290{"squared":true}
1296{"slope":0.5}
1299—
1302{"slope":0.5}
1303{"power":2}
1307{"slope":0.5}
1308{"mlp_slope":0.5,"crawler_mlp_slope":0.5}
1310{"negative_slope":0.5}
1311{"slope":0.5,"squared":true}
1312{"negative_slope":0.5}
1313{"slope":0.5}
1318{"negative_slope":0.5}
1319{"layers":11,"model_dim":512,"mlp_expansion":3}
1320{"slope":0.5,"power":2}
1321{"slope":0.5}
1322—
1324{"squared":true,"negative_slope":0.5}
1325{"slope":0.5}
1328{"negative_slope":0.5}
1329{"negative_slope":0.5}
1330{"negative_slope":0.5}
1355—
1359{"slope":0.5}
1364{"mlp_multiplier":3}
1368{"slope":0.5,"squared":true}
1376{"slope":0.5,"mlp_multiplier":3}
1378{"variant":"squared"}
1379{"squared":true,"slope":0.5}
1383{"slope":0.5}
1384{"slope":0.5}
1386{"slope":0.5,"mult":3}
1389{"squared":true,"alpha":0.5}
1390{"negative_slope":0.5}
1395{"slope":0.5}
1399{"slope":0.5}
1400{"slope":0.5}
1405{"slope":0.5}
1406{"slope":0.5}
1407{"layers":11,"dimensions":512,"heads":8,"kv_heads":4}
1410{"negative_slope":0.5,"squared":true}
1413{"slope":0.5}
1414{"squared":true,"alpha":0.5}
1420{"negative_slope":0.5}
1424{"layers":11,"dimensions":512,"heads":8,"kv_heads":4}
1427{"negative_slope":0.75}
1431{"slope":0.5}
1437{"negative_slope":0.5}
1438{"squared":true,"slope":0.5}
1440{"slope":0.5}
1442{"mlp_multiplier":3}
1444{"slope":0.5}
1446{"squared":true,"negative_slope":0.5}
1450{"slope":0.5}
1453{"slope":0.5}
1456{"negative_slope":0.5}
1457—
1471{"slope":0.5}
1472{"negative_slope":0.5}
1473{"squared":true}
1484{"negative_slope":0.5,"squared":true}
1485{"slope":0.5,"squared":true}
1486{"negative_slope":0.5,"squared":true}
1492{"slope":0.5}
1493{"slope":0.5}
1494{"slope":0.5}
1499—
1507{"negative_slope":0.5}
1514{"slope":0.5}
1515{"slope":0.5}
1517—
1520{"negative_slope":0.5}
1523{"negative_slope":0.5}
1527{"slope":0.5}
1528{"slope":0.5,"squared":true}
1530{"slope":0.5}
1537{"slope":0.5}
1538{"slope":0.5}
1539{"negative_slope":0.5}
1540{"negative_slope":0.5}
1541{"slope":0.5,"power":2}
1546{"slope":0.5}
1548—
1549{"negative_slope":0.5}
1550{"slope":0.5}
1555{"negative_slope":0.5,"power":2}
1560{"slope":0.5}
1561{"slope":0.5}
1565{"squared":true,"negative_slope":0.5}
1568{"slope":0.5}
1573{"negative_slope":0.5}
1581{"slope":0.5}
1583{"slope":0.5}
1584{"slope":0.5}
1585{"slope":0.5}
1586{"slope":0.5}
1606{"negative_slope":0.5,"squared":true}
1616{"negative_slope":0.5}
1619{"squared":true,"negative_slope":0.5}
1621{"squared":true,"negative_slope":0.5}
1626{"slope":0.5}
1628{"slope":0.5}
1629{"slope":0.5}
1639{"slope":0.5}
1640{"alpha":0.5}
1643—
1646{"squared":true,"negative_slope":0.5}
1647{"slope":0.5}
1648{"layers":11}
1650{"negative_slope":0.5}
1655{"slope":0.5}
1660{"slope":0.5}
1667{"negative_slope":0.5}
1670{"slope":0.5}
1675{"negative_slope":0.5}
1676{"slope":0.5}
1679{"squared":true}
1688{"slope":0.5}
1689{"slope":0.5}
1693{"negative_slope":0.5}
1696{"mlp_multiplier":3}
1709{"negative_slope":0.5}
1714{"slope":0.5}
1715{"slope":0.5}
1716{"slope":0.5}
1720{"slope":0.5}
1731{"slope":0.5}
1737{"negative_slope":0.5}
1754{"squared":true,"negative_slope":0.5}
1755{"negative_slope":0.5}
1759{"slope":0.5}
1760{"slope":0.5,"mlp_multiplier":3}