Emerging Techniques

Methods not yet mapped to a deep dive concept. These are new or uncommon techniques worth watching.

1022 unmapped methods

ArchitectureXSA

ArchitectureMLP3x

ArchitecturePartial RoPE

ArchitectureLeakyReLU

Architectureweight tying

Sequence Length2048

Test-Time Trainingscore-first TTT

ArchitectureKV head count

ArchitectureGQA

Compressionlzma

Architecturedepth recurrence

Architecturetied embeddings

Compressionzlib

Quantizationint6

ArchitectureRoPE

Regularizationlogit softcap

RegularizationLN scale

Regularizationlayerwise LN scale

QuantizationQAT

Test-Time Trainingfull TTT

LR Schedulecosine decay

ArchitectureVE128

OptimizerParallel Muon

QuantizationSTE QAT

QuantizationGPTQ-lite

Quantizationlate QAT

Sequence Lengthunknown

ArchitectureGated Attention

ArchitectureValue Residual

Weight AveragingEMA + SWA

Regularizationmagnitude pruning

Compressionbrotli

Quantizationmixed int6/int8

Evaluationstride-based eval

ArchitectureReLU²

Regularizationgradient clipping

Architectureparallel residuals

Sequence Length4096

CompressionBrotli

ArchitectureTrigramHash

ArchitectureLN Scale

Weight AveragingEMA + Tight SWA

Quantizationfp16

Test-Time TrainingTTT

Quantizationint5

Sequence Length8192

Quantizationmixed int6

ArchitectureMLP

Architectureother

Initializationspectral init

ArchitectureSwiGLU

Weight AveragingTight SWA

Quantizationint6 QAT

ArchitectureMLP4x

Sequence Length32768

RegularizationLN Scale

LR Schedulewarmup

Initializationresid mix

OptimizerNorMuon

Compressioncustom

LR Schedulewarmup + warmdown

ArchitectureEMA

ArchitectureHybrid

ArchitectureValue Embeddings

InitializationOrthogonal init

ArchitectureSWA

ArchitectureValueEmbedding

ArchitectureGatedDeltaNet

ArchitectureQK-Gain

Sequence Length256

ArchitectureXSA4

ArchitectureLeakyReLU²

ArchitectureMamba

ArchitectureEigenweight

Architecturedepth

ArchitectureVRL

ArchitectureOrthoInit

LR Schedulelinear warmup

ArchitectureTransformer

Regularizationdropout

Test-Time TrainingAdamW TTT

Architectureattention

ArchitectureGated DeltaNet

ArchitectureTransformer depth

Evaluationlong context eval

OptimizerMuon + Adam

LR Schedulecosine warmdown

ArchitectureMLP3.5x

Regularizationpruning

Regularizationlabel smoothing

Weight AveragingPolyak averaging

ArchitectureMLP activation

ArchitectureLoRA

Architectureresidual mixing

Architecturelogit softcap

Quantizationmixed int5/int6/int8

Initializationorthogonal init

Quantizationint4

ArchitectureShared Value Embedding

ArchitectureLeakyReLU(0.5)^2

Sequence Length512

QuantizationSTE QAT int6

ArchitectureSwiGLU MLP

Quantizationmixed int5/int6 QAT

ArchitectureValueResidual

Quantizationmixed int6/int5

Test-Time Trainingnone

OptimizerMuon + AdamW

Sequence Length131072

ArchitectureMLP expansion

Architecturesliding window eval

ArchitectureParallel Residuals

Architecturesliding window attention

Sequence Length16384

Architectureskip connections

Quantizationternary

Evaluationstride-based sliding window eval

InitializationOrthogonal

ArchitectureValue Embedding

LR Schedulewarmdown3500

ArchitectureU-Net

ArchitectureXSA-all

CompressionLZMA

RegularizationCROWN-Q penalty

Evaluationmulti-order n-gram backoff

ArchitectureGELU pre-enrichment

Regularizationz-loss

Sequence Length1536

Architectureattention modifications

ArchitectureDeltaNet

Sequence Length32000

Architectureattention modification

ArchitectureMLP hidden size

Architectureiteration embeddings

LR Schedulelinear warmup + warmdown

ArchitectureRMSNorm

LR Schedulecosine decay with linear warmup

ArchitectureCross-Repeat Skip

Quantizationmixed int8/int6

Initializationovertone embedding init

Initializationovertone init

InitializationOvertone init

InitializationOvertoneInit

Quantizationmixed int5/int6/int7

ArchitectureMLA

Quantizationmixed int4/int5

Test-Time Trainingtwo-phase TTT

LR Schedulelinear warmdown

ArchitectureBigramLogitHead

ArchitectureMoE

Architecturelayerwise LN scale

Quantizationint5/int6

ArchitectureMLP width

ArchitectureFlashAttention-3

Test-Time TrainingLegal TTT

RegularizationSIGReg

ArchitectureXSA-4

ArchitectureValue Residual Learning

QuantizationFP8

ArchitectureLeakyReLU^2

ArchitectureEBLS

ArchitectureLeakyReLU(0.9)^2

EvaluationkNN-LM

Regularizationratio loss

Sequence Length6144

Test-Time TrainingSLOT

Quantizationint7

OptimizerMuonEq-R

Initializationlinear scale init

ArchitectureParallel residuals

InitializationQK gain init

Evaluationerror correction table

ArchitectureQK-norm

Weight AveragingLAWA

ArchitectureBitLinear

ArchitectureLoop Embedding

LR Schedulewarmup and warmdown

Architecturephase-transition residual mixing

Test-Time Trainingonline logit bias

OptimizerMuonAdamW

ArchitectureMixture of Softmax

ArchitectureBackout

Architectureloop embeddings

Architectureresid_mix

Architecturememory tokens

ArchitectureMemory Tokens

Test-Time Trainingcausal TTT

RegularizationL1 regularization

ArchitectureParameter Banking

ArchitectureMLP4

Architectureactivation

Test-Time Trainingunknown

QuantizationLate QAT

ArchitectureSwiGLU FFN

LR Schedulewarmdown + cosine decay

Architecturedepth recurrence / weight sharing

ArchitectureGEPA

Architecturelayers

Initializationzero-init

ArchitectureU-Net Skip Gates

ArchitectureSmearGate + BigramHash

ArchitectureValue Residual (ResFormer)

Test-Time TrainingSGD TTT

QuantizationFull GPTQ

QuantizationQAT-export alignment

ArchitectureTied embeddings

Quantizationint5 GPTQ

Test-Time Trainingscore-first AdamW TTT

QuantizationINT6 QAT

ArchitectureLeakyReLU(0.5)^2 activation

Architecturedepth-scaled residual

ArchitectureValue Residual Learning (VRL)

Quantizationint8 QAT

Quantizationint6 QAT + GPTQ

ArchitectureSharedSparseSidecar

QuantizationFP16

ArchitectureBlock Attention Residuals

ArchitectureActivation

Evaluationmin-NLL epoch selection

Quantizationmixed int6 GPTQ

Quantizationint6 per-row with GPTQ-lite clip search

ArchitectureTied Embeddings

Regularizationweight pruning

Quantizationint6 per-row

ArchitectureU-Net encoder/decoder

ArchitectureFactored tied embedding

OptimizerNeoMuon

Evaluationonline n-gram cache eval

Test-Time TrainingTTT disabled

Quantizationint5 QAT

Quantizationmixed int5/int8

ArchitectureASQU

QuantizationQAT + GPTQ

Weight AveragingSWA + EMA

Test-Time Trainingdisabled

LR ScheduleLR scheduling tuned for single-device run

Sequence Length131000

Architecturebigram embedding guard

LR Scheduleadaptive cosine decay

LR ScheduleWarmup-Stable-Decay cosine schedule

Regularizationweight tying

ArchitectureBackoffNgramMixer

Architecturebidirectional transformer

Architecturevalue embeddings

Architectureloop embedding

Evaluationn-gram backoff cache

Evaluationtwo-pass n-gram rescoring

RegularizationCROWN-Q

ArchitectureDenseFormer

Evaluationfull-rescore

Architecturepredictor MLP

Evaluationtemperature sharpening

Evaluationtemperature scaling

Architectureshort convolution

ArchitectureJEPA

QuantizationINT6

ArchitectureSSM

Regularizationstructured pruning

ArchitectureFiLM

Architectureshared attention

Architecturebidirectional attention

Evaluationn-gram cache

ArchitectureEngramLite

Quantizationmixed int6/int7/int5

Test-Time TrainingAdam TTT

ArchitectureUnified Attention

Quantizationmixed int6/int4

Architectureerror feedback

RegularizationJacobian proxy loss

Architectureattention projections

Sequence Length524288

ArchitectureSLOT

Compressionbrotli + lzma

ArchitectureSP2048 vocabulary

OptimizerL-BFGS

Quantizationint6_awq

Regularizationcomplementary training

ArchitectureGDN

Architecturepause tokens

ArchitectureBasis Sharing

ArchitectureUniversal Transformer + ACT

Evaluationstateful-overlap eval

Regularizationdispersion loss

Architecturedepth / layer count

OptimizerMuon/Adam

Architecturereduced depth

Regularizationresidual scaling

Evaluationtest-time compute scaling

Evaluationlogit chunking

EvaluationNTK-aware RoPE scaling

Quantizationmixed int6/int8 STE QAT

Quantizationmixed int6/fp16

Architecturedepth/narrow transformer

Architecturelayer recurrence

Quantizationmixed int8/fp16

Architecturevocab size

ArchitectureMLP2x

ArchitectureMTP auxiliary head

Architecturedepth/width tradeoff

Quantizationmixed int6 quantization

Architecturedepth reduction

Architecturelogit softcapping

Initializationproj zero-init

Initializationresid_mix

LR Schedulelinear warmup + constant + cosine cooldown

LR Schedulewarmdown with LR floor and cooldown fraction schedule

LR Schedulelinear warmup + wallclock-aware linear warmdown

Architecturelayer looping

ArchitectureNTK-RoPE

Test-Time Trainingdoc-isolated eval

ArchitectureTransformer layers

Initializationovertone spectral embedding initialization

Initializationphase-transition residual-mix initialization

ArchitectureGQA / KV head count

Architecturelinearized neural memory

Initializationovertone spectral embedding init

Architecturepre-enrichment block

Architecturewider-shallower Transformer

ArchitectureMLP2.75x

Regularizationgrad clip

Architecturepre-enrichment

LR Schedulewarmup/warmdown

ArchitectureCTM workspace bridge

Initializationq_gain init

ArchitectureFlashAttention 3

Evaluationdoc-isolated sliding window eval

Evaluationpartial-window fix

Quantizationint6 STE QAT

Architectureencoder-decoder skip connections

Quantizationternary VQ

Architecturelow-rank Q

ArchitectureQK-Norm

ArchitectureFlashAttention-2

ArchitectureLRU / state space model

Architectureparallel scan

Architecturegated projection

ArchitectureReLU^2 MLP

Architecturenum_layers

Architecturepersistent memory

Architecturelow-rank factorization

Architecture10-layer 4xMLP

Architecturephase-transition resid_mix

LR Scheduleextended warmup

Quantizationmixed-bit lowbit export

ArchitectureTransformer depth/width

Evaluationvalidity-safe eval path

Evaluationnon-overlapping final eval

Sequence Length768

ArchitectureTransformer size

Quantizationmixed selective precision

InitializationQK Gain Init

InitializationQK_GAIN_INIT

Test-Time Trainingtiny eval-time SGD

Architecturedepth sharing / shared-depth

ArchitectureRMSNorm interface

Architecturephase-conditioned scales

Evaluationeval-time probability blending / context mixing

Evaluationstandard eval

Evaluationint8+zlib roundtrip evaluation

Architecturetokenizer/vocabulary size

InitializationSVD spectral init

Regularizationcompression-aware auxiliary loss

Architecturelow-rank K projection

Architecturelow-rank TD projection

Architecturelow-rank GRU state carry

Evaluationneural cache

Regularization3% magnitude pruning

LR Schedulewarmdown cosine schedule

ArchitectureTransformer depth / tied embeddings / KV head count

Initializationspectral init / residual mixing

ArchitectureCanon ACD

ArchitectureLow-Rank Q

Architecture12 layers

Initializationovertone spectral init

Quantizationint6 mixed

LR Schedulefixed learning rates

Evaluationcross-window KV caching

Architectureloop gates

Initializationzero initialization for loop embeddings

Initializationuniform gate initialization

Architecturedepth recurrence / looped transformer

ArchitectureBigram features

ArchitecturePer-head temperature scaling

Compressionzstandard

ArchitectureLate-K FP16

InitializationOvertone SVD init

Architectureper-layer scalars

LR Schedulewarmup schedule

ArchitectureDifferential Attention V2

Architecturelow-rank Q delta

Architectureloop position embeddings

ArchitectureMLP width reduction

Architecture10L Transformer

Quantizationmixed int6 QAT

Quantizationternary QAT

ArchitectureMLP3.25x

Weight AveragingEMA/SWA

Quantizationmixed int6/int8 with STE

Architecture11-layer U-Net Transformer

Test-Time TrainingReptile meta-learning TTT

ArchitectureGQA + RoPE

ArchitectureINL BetaMu attention

ArchitectureSort-Split MoE

ArchitectureALiBi

ArchitectureToken-routed MoE

ArchitecturePID Dynamics / INL Ultra-Lite

LR Schedulecosine warm restarts (SGDR)

Test-Time Trainingself-distillation TTT

Architecturex0 residual mix

ArchitectureShared Value Embeddings

Architectureencoder-decoder depth split

Architecturelearned per-dimension control knobs

ArchitectureMLP3x/MLP4x

Initializationphase-transition residual mixing

Architecturedepth reduction / encoder-decoder split

Architectureper-dimension control parameters

ArchitectureLayerNorm scale

ArchitectureCANON

InitializationCANON delta gate near-identity init

Test-Time TrainingSelf-Distillation TTT

Quantizationmixed int6/int5 QAT

ArchitectureDiffTransformer V2

Architectureweight sharing / depth recurrence

ArchitectureMLP×5

Architecturebackout connection

Architectureper-head temperature

Initializationortho+muP init

Quantizationmixed int5/int6 with fp16 embeddings

Architecturedepth and MLP width increase

Architectureq_proj

Sequence Length960

LR Scheduleshort-to-full context warmup

ArchitectureRadial Token Branch

ArchitectureBitNet-style ternary projections

ArchitectureMLP3x/4x MLP

ArchitectureAuxNet

Architecturesmear transformation

Weight AveragingEMA-SWA

Architecturedepth increase

Architectureextra RMSNorm

Evaluationmanual logits-only exact evaluation

ArchitectureCatalytic Residual Connections

Test-Time TrainingSGD post-quantization

InitializationOrthogonal + muP-scaled init

Regularizationgrad_clip

Test-Time Trainingscore-first full-model TTT

LR Schedulewarmup + warmdown + cosine decay

Regularizationweight entropy regularization

ArchitectureKronecker attention

Architectureskip-gram hash

Regularizationentropy token masking

Architecturegrouped-query attention

Quantizationint6 + zstd

LR Schedulewarmup + warmdown cosine decay

Quantizationmixed int5

Evaluationscore-first per chunk evaluation

Architectureper-layer scaling

ArchitectureCatalytic Residuals

Quantizationmixed int6/int8 QAT

Quantizationint16

ArchitectureHECR quantum state vectors

Architecturemulti-kernel readout heads

ArchitectureComplexSSM

Architecturecausal self-attention

ArchitectureU-Net skip connection

Quantizationmixed int6/int5/int4

ArchitectureLate Soft-Round QAT

Test-Time Trainingscore-first TTT with EB-adaptive per-layer scaling

Quantizationmixed int5/int6/int7 QAT

RegularizationLN scale depth damping

ArchitectureSmearGate + BigramHash embeddings

Quantizationint8 with FP16 token embedding

Test-Time Trainingskipped

InitializationOrthogonal loop positions

Architecturedepth recurrence / recursive weight sharing

InitializationQR-initialized orthogonal loop position embeddings

ArchitectureBlock AttnRes

ArchitecturePhiAlpha Simple

ArchitectureMLP width multiplier

ArchitectureTrigramHashEmbedding

OptimizerAdamW with Muon

Quantizationint6 + GPTQ-lite + QAT

ArchitectureValue Embeddings (VE128)

Architecturegated U-Net skip connections

ArchitectureCatalytic residuals

ArchitectureValue residual (ResFormer)

Regularizationembedding freeze

ArchitectureU-Net gated skips

QuantizationInt6 QAT

Test-Time Trainingdelayed outside-context-only PPM

RegularizationEMA weights, LN Scale

Quantizationzstd

ArchitectureLayer-Norm Scale

Regularizationfreeze early layers

LR Schedulecustom tuning from multi-device to single-device scale

ArchitectureLayer count

ArchitectureEmbeddings

ArchitectureAttention

Quantizationmixed int8/fp16 with custom codebook quantization

ArchitectureLeakyReLU(0.5)² activation

ArchitectureLeakyReLU(0.5)² MLP

Quantizationfp8

ArchitectureDG Attention

ArchitectureFlash Attention

Quantizationmixed int6/int8 with GPTQ-lite

ArchitectureShared VE128

LR ScheduleLate QAT

QuantizationEarly QAT

LR Schedulewarmup + warmdown cosine schedule

LR Scheduleauto warmdown

ArchitectureFiLM conditioning

ArchitectureBigramHash + TrigramHash

ArchitectureKV heads

Compressioncustom packed_zstd

Architectureshared sparse sidecar

Architecturevalue residual

Architecturegated attention

ArchitectureMLP3x with LeakyReLU(0.5)^2

Initializationloop gates initialized at 1.0

Quantizationmixed int5 (MLP) / int6 (attention) + GPTQ-lite per-row clip search + 3% magnitude pruning + FP16 passthrough for embeddings + zstd-22 compression

Evaluationsliding window eval + Test-Time Training (TTT)

Evaluationscore every epoch

ArchitecturePartial RoPE + NTK-aware scaling

Quantization2% magnitude pruning post-quantization

ArchitectureTrigramHash Embedding

ArchitectureBigramHash Embedding

ArchitectureStar-ReLU

Test-Time Trainingscore-first multi-pass legal TTT

Architecturedepth recurrence, weight tying, tied embeddings, RoPE, ReLU² MLP 3×, GQA

Quantizationint5 QAT + GPTQ

Regularization2% pruning

Test-Time Trainingfull TTT with SGD

QuantizationGPTQ with early QAT

Test-Time TrainingLegal Score-First TTT

ArchitecturePartial RoPE, XSA, BigramHash, VE128, SmearGate, logit softcap, tied embeddings

Evaluationsliding window eval with stride 32

Quantizationint6 per-row with GPTQ Hessian-aware quantization

Architecturerecursive weight sharing

Architectureasymmetric weight sharing (Micro Crawler)

Architecturebidirectional persistent deliberation gate

Architectureinput conditioning

Architectureposition embeddings

OptimizerMuon (matrices) and AdamW (embeddings and scalars)

Weight AveragingSWA and EMA

Quantizationfull-run Int6 QAT with STE

Quantizationint5 quantization

Architectureregister tokens

Architecturegated V-norm

Architecturemixture of softmax

Quantizationint6 per-row with Hadamard rotation

ArchitectureShared Value Embeddings (VE128)

ArchitectureLayer Norm Scale

ArchitecturecuDNN SDPA

Quantizationmixed Int5/Int6 QAT

ArchitectureValue Embed

ArchitectureEmbedding

ArchitectureLN depth scaling

ArchitectureValue embeddings

ArchitectureLate QAT

ArchitectureHybrid Attention + Mamba SSM

OptimizerMuon (matrix), Adam (scalar/embed)

ArchitectureOrthogonal initialisation

ArchitectureBigram hash embeddings

ArchitectureGQA (Grouped-Query Attention)

QuantizationQAT int6

QuantizationSTE QAT (late QAT) + Full GPTQ + Int5 MLP re-quantization + GPTQ-lite

ArchitectureValue Residual (VR)

ArchitectureGated Attention (GA)

ArchitectureBigramHash embeddings

Test-Time TrainingSGD TTT (legal, cosine, per-layer)

ArchitecturePer-head gated attention

ArchitectureLooped middle blocks

ArchitectureSelective ±1 magnitude pruning

ArchitectureLeakyReLU(0.5)² MLP 3x

QuantizationFull Hessian GPTQ

ArchitecturePartialRoPE

ArchitectureLNScale

ArchitectureValueEmbed

ArchitectureLateQAT

ArchitectureK projection LoRA

Quantizationint6 per-row with GPTQ-lite

ArchitectureK-Projection LoRA

ArchitectureResidual Input Mixing

Quantizationmixed int5/int6 with QAT

Architecturedepth-scaled residuals

Architecture11L Shared

Architectureskip_connections

Architecture1+7+1 layer stack

ArchitectureSolarShield gating

Test-Time TrainingNo TTT

Regularizationfreeze early layers during TTT

ArchitectureAttention-Residuals

QuantizationINT6 GPTQ-lite

QuantizationFull Hessian GPTQ with amax-aligned QAT

Quantizationint6 uniform + GPTQ-lite

ArchitectureMLP 3.5x with LeakyReLU(0.5)^2

ArchitectureXSA all 11 layers

ArchitectureTied FP16 embeddings

OptimizerAdam-style groups

Weight AveragingSWA+EMA blend

QuantizationBitNet b1.58 ternary quantisation with FP8 QAT

ArchitectureFused QKV projection

CompressionBase-3 + LZMA

RegularizationZ-loss regularisation

Quantization1-bit binary quantisation

ArchitectureYaRN positional encoding

Compressionbit-packing + LZMA

RegularizationPolynomial softcap with Z-loss regularisation

OptimizerMuon and Adam for training; SGD with momentum for TTT

LR Schedulecosine warmdown with linear warmup

Regularizationweight decay and layerwise LN scale

InitializationmuP scaling

ArchitectureOLR-FW

LR Schedulebeta2 decay

LR Schedulelearning rate scaling

Architecturelayerwise residual mixing

ArchitectureLN scaling

ArchitectureHybrid GDN/Transformer

Quantizationplanned but not implemented

ArchitectureMish² Activation

ArchitectureLayerNorm Scale

OptimizerParameter Banking + Parallel Muon

ArchitectureBigram Vocab

ArchitectureMLP 3×

ArchitectureSoft MoE

Test-Time Trainingstreaming legal TTT

Architecturemanifold-guided token interaction graph

Architecturesparsemax routing

Architecturespectrally-modulated gated hop cells

Architecturemanifold-guided attention

Architectureparallel transport across token manifold

LR Schedulecosine decay + hold + linear warmdown

Initializationdeterministic physics simulation initialization

Architecturespiking MLP

Regularizationspike-rate regularization

ArchitectureTRN hybrid

QuantizationSpinQuant/Hadamard

QuantizationSoft-Round QAT

Quantizationselective pruning

ArchitectureQKV fusion

ArchitectureLeakyReLU² stack

ArchitectureKV GQA

Test-Time Trainingvalidation set training

InitializationSVD-based attention warm-start

Architectureshared last layer

ArchitectureShort Conv

ArchitectureMoC

ArchitectureBankedLinear

ArchitectureMLP expansion adjustment

Initializationdepth-aware initialization

Architecturewavelet-lite mixer

ArchitectureTTT disabled

Initializationwavelet init

QuantizationGPTQ-lite int6

ArchitectureAttention shift mixing

ArchitectureK gain

ArchitectureLocal value residual

ArchitectureJEPA encoder-decoder

Sequence Length2047

LR Schedulecosine recovery

OptimizerMuon/AdamW

ArchitectureGPTQ-lite

ArchitectureCache+Backout

ArchitectureU-Net style skip connections

Evaluation5-gram eval interpolation

ArchitectureLeakyReLU2

ArchitectureGQA attention

LR Schedulestandard LR scheduling tuned for single-device run

ArchitectureMiddle-Out Autoregressive Compressor (MOAC)

Evaluationsliding window eval with backward-looking 7-gram cache

Test-Time Trainingscore-first TTT-like cache update

ArchitectureCastedLinear clip factor estimator

Architecture11L Transformer

Evaluationn-gram eval cache

Initializationones-init

ArchitectureHedge Mixer

Quantizationmixed FP4/Int6 QAT

InitializationDeepNorm init

RegularizationZ-Loss

RegularizationQK-Clip

Evaluationonline n-gram cache

Evaluationmulti-order n-gram cache interpolation

ArchitectureCROWN-Q

ArchitecturePairHash

Evaluationfull validation on fineweb_val_* split

Evaluationmulti-order backoff n-gram eval

Evaluationadaptive alpha evaluation

Test-Time TrainingMLP-down-only TTT

Test-Time TrainingMLP-all TTT

QuantizationInt6 STE QAT

Test-Time Trainingscore-first full TTT

ArchitectureLeakyReLU(0.5)^2 MLP

RegularizationLN scaling

Evaluationbackward-looking eval cache

Architecture8-layer architecture

ArchitectureLeakyReLU^2 MLP

Evaluationmulti-order backoff n-gram eval cache

Evaluation7-gram backoff

Evaluationadaptive n-gram backoff eval

ArchitectureLeakyReLU(0.9)²

Evaluationmulti-order n-gram eval

ArchitectureN-gram cache

Test-Time TrainingCosine TTT

Evaluationentropy-adaptive cache blending

LR Schedulenone

Evaluationlegal score-first 7-gram backoff

ArchitectureLeakyReLU² MLP

Evaluationorder-adaptive entropy-gated n-gram backoff cache

Evaluationn-gram backoff

ArchitectureU-Net-style skip structure

Evaluationorder-adaptive n-gram backoff cache

Regularizationlayerwise LN scaling

Evaluation7-gram causal cache with entropy-adaptive blending

Evaluation7-gram n-gram cache

LR Scheduledynamic wallclock cosine warmdown

QuantizationSTE QAT / post-quant 6-bit

Evaluationentropy-adaptive alpha

LR Schedulelate QAT activation based on LR scale threshold

Evaluationmulti-order backoff n-gram cache

Evaluationdistributed cache pre-fill

Initializationasymmetric LoRA initialization

Evaluationfull evaluation

Evaluationmulti-GPU n-gram prefill

Evaluationchunk-based sequential evaluation

Test-Time Trainingscore-first TTT-like n-gram cache

ArchitectureBankLinear

Initializationdepth-aware mixing coefficient initialization

Architecturelarger MLP

ArchitectureLeakyReLU MLP

Initializationwarm-start cubric initialization

ArchitectureHWNODE

Architecturespectral normalization

ArchitectureadaLN timestep conditioning

Evaluationvariational bound evaluation with discrete absorbing-mask process

Architecturelearned level signals

ArchitectureGatedAttn

ArchitectureXSA6

ArchitectureBigramHash4K

Test-Time Traininglegal TTT

Evaluationorder-adaptive entropy-gated BackoffNgramMixer

Evaluationint8+zlib roundtrip eval

ArchitectureByte-level transformer

ArchitectureJEPA auxiliary loss

ArchitectureLinear gate head

ArchitectureLeakyReLU_LegalTTT_ParallelMuon

Evaluationfine-grained n-gram cache chunked evaluation

Test-Time Trainingscore-first legal TTT

ArchitectureGPT depth increase

ArchitectureMLP_MULT reduction

ArchitectureBigram embedding modification

ArchitectureToken embedding / VE dimension reduction

Test-Time TrainingLegalTTT

ArchitectureGatedAttention

ArchitectureU-Net encoder-decoder

RegularizationLate QAT soft-round STE

Evaluationorder-9 n-gram backoff cache

RegularizationEMA

ArchitectureSelective Scan (Mamba)

Evaluationn-gram backoff with extended order

Evaluationlarger chunked cache refresh

ArchitectureOutput-LN

ArchitectureBirkhoff mixing

Architecturetimestep scaling

Architecturecross-repeat skip

Architecturelearned mixer head

Architecturefrozen n-gram oracle

ArchitectureMHA 8/8

Evaluationscore-first backward-looking n-gram cache

LR Schedulematrix learning rate tuning

Evaluationscore-first eval

Evaluationscore-first n-gram backoff

Evaluationvectorized 7-gram backoff + kNN-LM

ArchitectureRandomLinearWithAdapter

Regularizationentropy-reg QAT

ArchitectureParallel Muon

Regularizationsemantic tube regularization

ArchitectureQK RMSNorm

RegularizationVICReg

Architectureprojection heads

Architecturedoc_copy_ctx2

Sequence Length16300000

ArchitectureVE64

Evaluationfull-rescore n-gram cache

Architecturedifferential attention

Evaluationtwo-pass full-rescore

Evaluationfull validation eval

ArchitecturePacked causal memory

Evaluationscore-first causal evaluation

Evaluationtokenizer-agnostic val_bpb evaluation

Evaluationfull-rescore two-pass N-gram

Evaluationorder-12 n-gram cache

Evaluationlong phrase cache

Evaluation65K chunking

Evaluation11-gram eval cache

ArchitectureKGIIR

Evaluationscore-first evaluation

Evaluationsingle-pass eval

Regularizationtemperature sharpening

Evaluationtwo-pass full rescore

Evaluationexact post-quant eval

ArchitectureLogisticContextMixer

QuantizationINT4

Architectureresid mix

Regularizationlogit bias

Initializationphase-mix init

EvaluationMC Dropout ensembling

ArchitectureMTP

Architectureanti-layer removal

ArchitectureLegal TTT

ArchitectureMHA

LR Schedulelinear warmup + cosine decay

ArchitectureBoxIntersectionMixer

ArchitectureGPT

Architecturevocab_bias

Architecturebias to pre-norms

LR Schedulelayer/depth schedule

QuantizationGPTQ mixed int6/int7

ArchitectureMTP heads

Evaluationcausal sequential chunk eval

ArchitectureJEPA bottleneck

ArchitectureadaLN

Architecturefrozen visible-token logits

Evaluationdiscrete absorbing-mask ELBO

Regularizationloss truncation

Evaluationdiscrete ELBO eval

Architectureiteration scales

ArchitectureHybridNorm

ArchitectureDifferential Attention

ArchitectureWaveletGPT

ArchitectureVGA

ArchitectureMulti-Token Prediction

EvaluationTurboQuant KV cache compression

Architecturetoken-shift mixing

Architectureattention window

ArchitectureFrozenRandomLinearWithLoRA

Sequence Length9000

Architecturehierarchical token processing

Architectureqk_gain

ArchitectureTurbo-Muon

ArchitectureMimetic V-O initialization

ArchitectureResidual lambdas

ArchitectureVE196

ArchitectureCache + backout

ArchitectureSSSL

Sequence Length448

Evaluationonline n-gram agreement eval

Evaluationautoregressive KV-cache eval

Evaluationautoregressive eval

Quantizationpolar

Test-Time Trainingrandom-map TTT

Test-Time TrainingTTT-Linear

ArchitectureFlowRefiner

Architecturehierarchical chunking

Architecturemulti-resolution processing

ArchitectureNativeFlowMatcher

Compressionbyte-shuffle

Architecturedepthwise Conv1D

ArchitectureDynamicChunker

EvaluationTriton eval kernels

Architecturecrawler bottleneck

Architectureshared TAP encoder connections

CompressionrANS

ArchitectureCausal n-gram fix

ArchitectureTRN

Evaluationentropy-adaptive mixing

ArchitectureMonarch Matrices

Architecturemini-MoE

Architecturelogit bias

ArchitectureResidualScale

Regularizationfocal loss

ArchitecturePRP

Evaluationtwo-pass eval

ArchitectureWARP-Len

ArchitectureWARP-Pos

ArchitectureWARP-Type

RegularizationLeakyReLU

Quantizationmixed int5/int6 GPTQ

ArchitectureFA3

ArchitectureTTT

ArchitectureMLP adapters

LR Schedulehold-cosine

Test-Time TrainingContext-Only SLOT

QuantizationQ-LoRA

Architecturecoordinate embeddings

Evaluationonline n-gram agreement

LR Schedulesplit-LR

ArchitectureH-Net

Evaluationexact sequence matching

ArchitectureLatentPredictor

Initializationdepth-aware init

Regularizationadaptive focal loss

Quantizationmixed int8/int7

LR Schedulelinear decay

Architecturestep embedding

Quantizationmixed int4/int8

Architecturemulti-model single representation

Architectureconv kernel

QuantizationProxQuant

ArchitectureSpiking-MLP

ArchitectureRBF

ArchitectureArcTan surrogate gradients

Architecturehomeostatic threshold adaptation

Architectureiter_embed

Architectureiter_gate

Regularizationrepeat penalty

Test-Time TrainingL-BFGS Causal SLOT

Test-Time TrainingCascaded 2-Phase L-BFGS

Test-Time TrainingDiscriminative per-block pre-quant TTT

Sequence Length128

Architectureencoder-decoder split

Test-Time TrainingFiLM-only TTT

Regularizationcompression-aware regularization

Architecturesignsq

QuantizationBF16 scales

Quantizationactivation binarization

ArchitectureSentencePiece 4096

ArchitectureSP4096

Evaluationquadrature over mask ratios

RegularizationSDClip

LR Schedulehigher LR compensation

ArchitectureGELU

Initializationinit_std

ArchitectureHadamard rotation

Evaluationn-gram tilt

Quantizationcodebook quant

RegularizationL2 loss

Architectureparallel blocks

Architecturedecoder depth

Architecturemodel width

OptimizerMousse

Quantizationmixed Q4/Q5/Q6

Architecturebyte-level input

Initializationlinear-by-depth scale init

Compressionint8

Quantizationmixed int7/int5

Evaluationhedge mixer

Quantizationmixed int6/int5/int4/fp16

ArchitectureDirectionalSemanticVec

ArchitectureHadamard Matrix

Evaluationmulti-seed evaluation

ArchitectureParallel Residual

Architecturedepth embeddings

Architectureweight scaling

ArchitectureSP8192

EvaluationBOS-reset non-overlap eval

Initializationlinear phase initialization

Initializationdepth-aware constant scale init

Evaluationfull validation comparison

Test-Time Trainingscore-first SLOT

ArchitectureJEPA-style regression transformer

QuantizationLeanICQ int3

QuantizationICQuant

RegularizationHessian clipping

EvaluationTap-In V6 cross-window

Regularizationloss weighting

ArchitectureV22

Compressionpyminify

ArchitectureTAP

ArchitectureANCHOR

Architectureparallel residual routing

LR Schedulebudget annealing

Regularizationspectral floor

RegularizationHessian-Aware SDClip

ArchitectureLoRA TTT

Architectureweight banking

Architecturehash embedding

ArchitectureSP1024 tokenizer

ArchitectureGated DeltaNet hybrid

ArchitectureLN scale

LR Schedulewarmup + stable + cosine decay

Regularizationgradient checkpointing

ArchitectureGDN-Hybrid

ArchitectureD-TPA

Sequence Length65536

ArchitectureVarLen Attention

Regularizationadaptive clip

RegularizationTWEO

Quantizationpost-training quantization

Regularizationskip gates

ArchitectureFMN

ArchitectureSparseBraidRegister

OptimizerFMNRiemannianAdam

Test-Time Trainingscore-first SGD

Architecturetrajectory-state readout

Test-Time TrainingqTTT

CompressionANS + brotli

Architecturerandom basis MLP

Architectureparallel residual lanes

ArchitectureQK depth ramp

ArchitectureMTP head

LR Schedulelinear cooldown

ArchitectureAttention Output Gate

Test-Time TrainingMP-SGD-TTT

ArchitectureK_KVShare_Wider

Quantizationfp8 e4m3

Evaluationchain-rule eval

LR Schedulelate loop onset

Test-Time Trainingreadout_only

ArchitectureGatedDeltaNet / Flash Linear Attention

Quantizationmixed int6/int7

Architecturebreadcrumb gating

Regularizationstochastic depth

Sequence Length786432

Quantizationmixed int4/int6/int8

ArchitectureQK gain

Quantizationmixed int8/int6/int4

QuantizationAWQ

Evaluationeval-only quantized path

Evaluationquantized-eval-only

InitializationQK gain

Architecturefactorized late layers

LR Schedulehealing phase

Initializationcustom random init

Evaluationfixed-depth eval