Activation Functions

Edit

General • 74 methods

Activation functions are functions that we apply in neural networks after (typically) applying an affine transformation combining weights and input features. They are typically non-linear functions. The rectified linear unit, or ReLU, has been the most popular in the past decade, although the choice is architecture dependent and many alternatives have emerged in recent years. In this section, you will find a constantly updating list of activation functions.

Subcategories

1 Adaptive Activation Functions

Methods

Add a Method

Method	Year	Papers
ReLU	2000	9438
GELU Gaussian Error Linear Units (GELUs)	2016	8152
Sigmoid Activation	2000	6259
Tanh Activation	2000	5664
Leaky ReLU	2014	1180
GLU Language Modeling with Gated Convolutional Networks	2016	626
Swish Searching for Activation Functions	2017	327
Softplus	2000	236
SELU Self-Normalizing Neural Networks	2017	225
Mish Mish: A Self Regularized Non-Monotonic Activation Function	2019	208
PReLU Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification	2015	100
ReLU6 MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications	2017	77
Hard Swish Searching for MobileNetV3	2019	73
FEM Features Understanding in 3D CNNs for Actions Recognition in Video	2020	55
LEAF Learnable Extended Activation Function (LEAF) for Deep Neural Networks	2023	52
Maxout Maxout Networks	2013	48
ELU Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)	2015	36
RBF	2000	34
SiLU Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning	2017	19
Shifted Softplus SchNet: A continuous-filter convolutional neural network for modeling quantum interactions	2017	14
ARiA ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets	2018	12
Softsign Activation	2000	12
Squared ReLU Primer: Searching for Efficient Transformers for Language Modeling	2021	10
CReLU Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units	2016	9
ASU ArXiving Before Submission Helps Everyone	2020	8
KAF Kafnets: kernel-based non-parametric activation functions for neural networks	2017	7
SwiGLU GLU Variants Improve Transformer	2020	6
TanhExp TanhExp: A Smooth Activation Function with High Convergence Speed for Lightweight Neural Networks	2020	5
modReLU Unitary Evolution Recurrent Neural Networks	2015	5
Poly	2000	5
Siren Implicit Neural Representations with Periodic Activation Functions	2020	4
Hard Sigmoid BinaryConnect: Training Deep Neural Networks with binary weights during propagations	2015	4
m-arcsinh m-arcsinh: An Efficient and Reliable Function for SVM and MLP in scikit-learn	2020	3
RReLU Empirical Evaluation of Rectified Activations in Convolutional Network	2015	3
Serf SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function	2021	3
Rational Activation function Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks	2019	3
DELU Trainable Activations for Image Classification	2023	2
SReLU Deep Learning with S-shaped Rectified Linear Activation Units	2015	2
NFN Neo-fuzzy-neuron based new approach to system modeling, with application to actual system	1994	2
PAU Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks	2019	2
Hermite Activation	2019	2
AHAF Adaptive hybrid activation function for deep neural networks	2022	2
Rational Activation Function Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks	2019	2
GCU Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks	2021	2
ReGLU GLU Variants Improve Transformer	2020	2
PELU Parametric Exponential Linear Unit for Deep Convolutional Neural Networks	2016	2
E-swish E-swish: Adjusting Activations to Different Network Depths	2018	2
Margin ReLU A Comprehensive Overhaul of Feature Distillation	2019	1
CosLU Trainable Activations for Image Classification	2023	1
SERLU Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)	2018	1
ShiLU Trainable Activations for Image Classification	2023	1
CoLU Deeper Learning with CoLU Activation	2021	1
GeGLU GLU Variants Improve Transformer	2020	1
CELU	2000	1
ReLUN Trainable Activations for Image Classification	2023	1
SSFG regularization SSFG: Stochastically Scaling Features and Gradients for Regularizing Graph Convolutional Networks	2021	1
Gumbel Activation Long-tailed Instance Segmentation using Gumbel Optimized Loss	2022	1
ScaledSoftSign Trainable Activations for Image Classification	2023	1
NormLinComb Trainable Activations for Image Classification	2023	1
NLSIG	2000	1
Phish Phish: A Novel Hyper-Optimizable Activation Function	2021	1
Smish	2000	1
ELiSH The Quest for the Golden Activation Function	2018	1
EvoNorms Evolving Normalization-Activation Layers	2020	1
Smooth Step The Tree Ensemble Layer: Differentiability meets Conditional Computation	2020	1
HardELiSH The Quest for the Golden Activation Function	2018	1
LinComb Trainable Activations for Image Classification	2023	1
Nipuna 0/1 Deep Neural Networks via Block Coordinate Descent	2022	1
StarReLU MetaFormer Baselines for Vision	2022	1
Lecun's Tanh	1998	0
ASAF Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization	2020	0
Hardtanh Activation	2000	0

Activation Functions Edit

Methods Add a Method

Activation Functions

Edit

Methods

Add a Method