Stochastic Optimization

Edit

General • Optimization • 62 methods

Stochastic Optimization methods are used to optimize neural networks. We typically take a mini-batch of data, hence 'stochastic', and perform a type of gradient descent with this minibatch. Below you can find a continuously updating list of stochastic optimization algorithms.

Subcategories

1 Large Batch Optimization

2 Momentum Rules

Methods

Add a Method

Method	Year	Papers
Adam Adam: A Method for Stochastic Optimization	2014	17365
SGD	1951	1737
Adafactor Adafactor: Adaptive Learning Rates with Sublinear Memory Cost	2018	575
RMSProp	2013	428
LAMB Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	2019	179
AdaGrad	2011	165
AdamW Decoupled Weight Decay Regularization	2017	146
SGD with Momentum	1999	141
Deep Ensembles Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles	2016	140
Gravity Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning	2021	132
MAS Mixing ADAM and SGD: a Combined Optimization Method	2020	92
FA Random feedback weights support learning in deep neural networks	2014	91
LARS Large Batch Training of Convolutional Networks	2017	66
Local SGD Local SGD Converges Fast and Communicates Little	2018	58
RAdam On the Variance of the Adaptive Learning Rate and Beyond	2019	50
AMSGrad On the Convergence of Adam and Beyond	2019	47
DFA Direct Feedback Alignment Provides Learning in Deep Neural Networks	2016	41
Apollo Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization	2020	38
Stochastic Weight Averaging Averaging Weights Leads to Wider Optima and Better Generalization	2018	37
Gradient Sparsification Gradient Sparsification for Communication-Efficient Distributed Optimization	2017	30
1-bit Adam 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed	2021	29
Nesterov Accelerated Gradient	1983	27
KP Deep Learning without Weight Transport	2019	25
Adabelief AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients	2020	18
Lookahead Lookahead Optimizer: k steps forward, 1 step back	2019	16
SMA	2000	15
AdaDelta ADADELTA: An Adaptive Learning Rate Method	2012	15
AdaBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	11
HGS	2000	10
Gradient Checkpointing Training Deep Nets with Sublinear Memory Cost	2016	10
Demon Demon: Improved Neural Network Training with Momentum Decay	2019	10
1-bit LAMB 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed	2021	9
Forward gradient Gradients without Backpropagation	2022	9
AdaMax Adam: A Method for Stochastic Optimization	2014	8
NT-ASGD Regularizing and Optimizing LSTM Language Models	2017	6
SM3 Memory Efficient Adaptive Optimization	2019	5
Distributed Shampoo Towards Practical Second Order Optimization for Deep Learning	2021	4
NADAM	2015	4
AdaShift AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods	2018	3
PowerSGD PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization	2019	3
gCANS Adaptive shot allocation for fast convergence in variational quantum algorithms	2021	2
QHM Quasi-hyperbolic momentum and Adam for deep learning	2018	2
Polyak Averaging	1991	2
MPSO Motion-Encoded Particle Swarm Optimization for Moving Target Search Using UAVs	2020	2
AdaSmooth AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio	2022	2
AMSBound Adaptive Gradient Methods with Dynamic Bound of Learning Rate	2019	1
Demon CM Demon: Improved Neural Network Training with Momentum Decay	2019	1
SLAMB SLAMB: Accelerated Large Batch Training with Sparse Communication	2023	1
SRMM Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates	2022	1
AdaMod An Adaptive and Momental Bound Method for Stochastic Learning	2019	1
AdaSqrt Second-order Information in First-order Optimization Methods	2019	1
DSPT Adaptive hybrid activation function for deep neural networks	2022	1
Powerpropagation Powerpropagation: A sparsity inducing weight reparameterisation	2021	1
SGDW Decoupled Weight Decay Regularization	2017	1
Demon ADAM Demon: Improved Neural Network Training with Momentum Decay	2019	1
YellowFin YellowFin and the Art of Momentum Tuning	2017	1
AggMo Aggregated Momentum: Stability Through Passive Damping	2018	1
ATMO Combining Optimization Methods Using an Adaptive Meta Optimizer	2021	1
MADGRAD Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization	2021	1
QHAdam Quasi-hyperbolic momentum and Adam for deep learning	2018	1
FASFA	2000	0

Stochastic Optimization Edit

Methods Add a Method

Stochastic Optimization

Edit

Methods

Add a Method