Documentation

Thresholds each element of the input Tensor.

Applies the rectified linear unit function element-wise, that is, \[ \text{ReLU}(x) = max(0, x). \]

Applies the gaussian error linear unit function element-wise.

Applies the gaussian error linear unit function element-wise.

This is the implementation of the GELU activation function from Google's BERT repo (and coincidentally also from OpenAI's GPT). See also https://arxiv.org/abs/1606.08415.

>>> t <- sFull (TensorSpec (SGradient SWithGradient) (SLayout SDense) (SDevice SCPU) (SDataType SFloat) (SShape $ SNil)) 0.5
>>> t' <- geluNew t
>>> fromTensor @Float t'
0.345714

Applies the HardTanh function element-wise.

Applies the hardswish function element-wise.

Applies the exponential linear unit function element-wise, with alpha input, \[ \text{ELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x) - 1)). \]

Applies the scaled exponential linear unit function element-wise, that is, \[ \text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)), \] with \(\alpha = 1.6732632423543772848170429916717\) and \(\text{scale}=1.0507009873554804934193349852946\).

Applies the continuously differentiable exponential linear unit function element-wise, that is, \[ \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)). \]

Applies the element-wise function: \[ \text{LeakyReLU}(x) = \max(0,x) + \text{negativeSlope} * \min(0,x), \] the the angle of the negative slope can be controlled. A typical value for it is 0.01.

Applies the parameterized rectified linear unit function element-wise, that is, \[ \text{PReLU}(x) = max(0, x) + \text{weight} * min(0, x). \] The weight parameter is typically learnable.