Leaky ReLU

Leaky ReLUs allow a small, positive gradient when the unit is not active,^[12] helping to mitigate the vanishing gradient problem.

Parametric ReLU

Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural-network parameters.^[17]

Note that for a ≤ 1, this is equivalent to

${\displaystyle f(x)=\max(x,ax)}$

and thus has a relation to "maxout" networks.^[17]

Gaussian-error linear unit (GELU)

GELU is a smooth approximation to the rectifier:

where ${\displaystyle \Phi (x)=P(X\leqslant x)}$ is the cumulative distribution function of the standard normal distribution.

This activation function is illustrated in the figure at the start of this article. It has a "bump" to the left of x < 0 and serves as the default activation for models such as BERT.^[18]

SiLU

The SiLU (sigmoid linear unit) or swish function^[19] is another smooth approximation, first coined in the GELU paper:^[18]

where ${\displaystyle \operatorname {sigmoid} (x)}$ is the sigmoid function.

Softplus

A smooth approximation to the rectifier is the analytic function

which is called the softplus^[20][8] or SmoothReLU function.^[21] For large negative ${\displaystyle x}$ it is roughly ${\displaystyle \ln 1}$, so just above 0, while for large positive ${\displaystyle x}$ it is roughly ${\displaystyle \ln(e^{x})}$, so just above ${\displaystyle x}$.

This function could be approximated as: ${\displaystyle \ln \left(1+e^{x}\right)\approx {\begin{cases}\ln(2),\quad x=0,\\{\frac {x}{1-e^{-{\frac {x}{\ln(2)}}}}},\quad x\neq 0\end{cases}}}$

By making the change of variables ${\displaystyle x=y\ln(2)}$, this is equivalent to

${\displaystyle \log _{2}\left(1+2^{y}\right)\approx {\begin{cases}1,&y=0,\\{\frac {y}{1-e^{-y}}},&y\neq 0.\end{cases}}}$

A sharpness parameter ${\displaystyle k}$ may be included:

The derivative of softplus is the logistic function.

The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:

${\displaystyle \operatorname {LSE_{0}} ^{+}(x_{1},\dots ,x_{n}):=\operatorname {LSE} (0,x_{1},\dots ,x_{n})=\ln \left(1+e^{x_{1}}+\cdots +e^{x_{n}}\right).}$

The LogSumExp function is

${\displaystyle \operatorname {LSE} (x_{1},\dots ,x_{n})=\ln \left(e^{x_{1}}+\cdots +e^{x_{n}}\right),}$

and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

ELU

Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.^[22]

In these formulas, ${\displaystyle a}$ is a hyper-parameter to be tuned with the constraint ${\displaystyle a\geq 0}$.

The ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form ${\displaystyle f(x)=\max(-a,x)}$, given the same interpretation of ${\displaystyle a}$.

Mish

The mish function could also be used as a smooth approximation of the rectifier.^[19] It is defined as

${\displaystyle f(x)=x\tanh {\big (}\operatorname {softplus} (x){\big )},}$

where ${\displaystyle \tanh(x)}$ is the hyperbolic tangent, and ${\displaystyle \operatorname {softplus} (x)}$ is the softplus function.

Mish is non-monotonic and self-gated.^[23] It was inspired by Swish, itself a variant of ReLU.^[23]

Squareplus

Squareplus^[24] is the function

${\displaystyle \operatorname {squareplus} _{b}(x)={\frac {x+{\sqrt {x^{2}+b}}}{2}}}$

where ${\displaystyle b\geq 0}$ is a hyperparameter that determines the "size" of the curved region near ${\displaystyle x=0}$. (For example, letting ${\displaystyle b=0}$ yields ReLU, and letting ${\displaystyle b=4}$ yields the metallic mean function.) Squareplus shares many properties with softplus: It is monotonic, strictly positive, approaches 0 as ${\displaystyle x\to -\infty }$, approaches the identity as ${\displaystyle x\to +\infty }$, and is ${\displaystyle C^{\infty }}$ smooth. However, squareplus can be computed using only algebraic functions, making it well-suited for settings where computational resources or instruction sets are limited. Additionally, squareplus requires no special consideration to ensure numerical stability when ${\displaystyle x}$ is large.