Regularization

Edit

General • 58 methods

Regularization strategies are designed to reduce the test error of a machine learning algorithm, possibly at the expense of training error. Many different forms of regularization exist in the field of deep learning. Below you can find a constantly updating list of regularization strategies.

Subcategories

1 Parameter Norm Penalties

Methods

Add a Method

Method	Year	Papers
Dropout Dropout: A Simple Way to Prevent Neural Networks from Overfitting	2014	19779
Label Smoothing	1985	9808
Attention Dropout	2018	7807
Weight Decay	1943	7791
Entropy Regularization Asynchronous Methods for Deep Reinforcement Learning	2016	795
R1 Regularization Which Training Methods for GANs do actually Converge?	2018	469
Early Stopping	1995	380
Stochastic Depth Deep Networks with Stochastic Depth	2016	330
Path Length Regularization Analyzing and Improving the Image Quality of StyleGAN	2019	169
Variational Dropout A Theoretically Grounded Application of Dropout in Recurrent Neural Networks	2015	133
Off-Diagonal Orthogonal Regularization Large Scale GAN Training for High Fidelity Natural Image Synthesis	2018	130
DropBlock DropBlock: A regularization method for convolutional networks	2018	122
Target Policy Smoothing Addressing Function Approximation Error in Actor-Critic Methods	2018	91
DropConnect Regularization of Neural Networks using DropConnect	2013	78
L1 Regularization	1986	71
Embedding Dropout A Theoretically Grounded Application of Dropout in Recurrent Neural Networks	2015	58
ALS Efficient Model for Image Classification With Regularization Tricks	2020	58
PGM	2000	55
Activation Regularization Revisiting Activation Regularization for Language RNNs	2017	49
Temporal Activation Regularization Revisiting Activation Regularization for Language RNNs	2017	49
SpatialDropout Efficient Object Localization Using Convolutional Networks	2014	27
Zoneout Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations	2016	27
LCC Regularisation of Neural Networks by Enforcing Lipschitz Continuity	2018	26
Manifold Mixup Manifold Mixup: Better Representations by Interpolating Hidden States	2018	23
DropPath FractalNet: Ultra-Deep Neural Networks without Residuals	2016	22
Orthogonal Regularization Neural Photo Editing with Introspective Adversarial Networks	2016	20
ROME Locating and Editing Factual Associations in GPT	2022	18
GAN Feature Matching Improved Techniques for Training GANs	2016	15
Concrete Dropout Concrete Dropout	2017	14
SRN Stable Rank Normalization for Improved Generalization in Neural Networks and GANs	2019	13
SCN Suppressing Uncertainties for Large-Scale Facial Expression Recognition	2020	13
LayerScale Going deeper with Image Transformers	2021	10
Auxiliary Batch Normalization Adversarial Examples Improve Image Recognition	2019	10
Euclidean Norm Regularization Deep Compressed Sensing	2019	9
Adaptive Dropout Adaptive dropout for training deep neural networks	2013	7
Shake-Shake Regularization Shake-Shake regularization	2017	5
SVD Parameterization Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization	2018	4
FIERCE Preserving Fine-Grain Feature Information in Classification via Entropic Regularization	2022	4
ShakeDrop ShakeDrop Regularization for Deep Residual Learning	2018	3
GradDrop Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout	2020	3
Recurrent Dropout Recurrent Dropout without Memory Loss	2016	3
GMVAE Regularizing Transformers With Deep Probabilistic Layers	2021	3
Batch Nuclear-norm Maximization Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations	2020	3
LayerDrop Reducing Transformer Depth on Demand with Structured Dropout	2019	3
Discriminative Regularization Discriminative Regularization for Generative Models	2016	2
Targeted Dropout Learning Sparse Networks Using Targeted Dropout	2019	2
Fraternal Dropout Fraternal Dropout	2017	2
Early Dropout Dropout Reduces Underfitting	2023	2
ScheduledDropPath Learning Transferable Architectures for Scalable Image Recognition	2017	2
DropPathway Audiovisual SlowFast Networks for Video Recognition	2020	1
AutoDropout AutoDropout: Learning Dropout Patterns to Regularize Deep Networks	2021	1
Spectral Dropout Regularization of Deep Neural Networks with Spectral Dropout	2017	1
Weights Reset The Weights Reset Technique for Deep Neural Networks Implicit Regularization	2023	1
STTP Spectral Tensor Train Parameterization of Deep Learning Layers	2021	1
Checkerboard Dropout	2000	0
rnnDrop	2015	0

Regularization Edit

Methods Add a Method

Regularization

Edit

Methods

Add a Method