relu
Builtin index · Statistics & ML
result = relu(x)
Input: x · Output: result
What it is
Elementwise activation applied to a number, vector, or matrix x of any shape — relu clips negative values to zero, sigmoid squashes to (0, 1).
Overloads
One call shape — this function is not overloaded.
| Code | Input | Output | Description |
result = relu(x) | x number, vector, or matrixnot documented yet | result — | Returns a value of the same shape as x, differentiable through the tape. |
Examples
From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.
w = param([0.5, -0.3; 0.2, 0.8])
b = param([0.1, -0.1])
x = [1.0, 2.0; -0.5, 1.5] # two examples, two features each
z = dense(x, w, b)
a = relu(z)
p = sigmoid(a)
loss = sum(p .^ 2)
g = grad(loss, wrt = w)
print("relu: {round(a, 4)}")
print("sigmoid: {round(p, 4)}")
print("dL/dw: {round(g, 4)}")
tape_reset() # clear the tape before the next computation
scores = [2.0, 1.0, 0.1; 0.5, 0.5, 0.5]
probs = softmax_rows(scores)
print("softmax row 0: {round(probs[0, :], 4)}")
seed(3)
kept = dropout(ones(1, 8), 0.5)
print("dropout output: {kept}")
print("survivors: {sum(kept > 0)} of 8, each scaled by 1/(1-rate)")
output
relu: [1, 1.4; 0, 1.25] sigmoid: [0.7311, 0.8022; 0.5, 0.7773] dL/dw: [0.2875, 0.12; 0.5749, 0.9128] softmax row 0: [0.659, 0.2424, 0.0986] dropout output: [2, 2, 0, 0, 2, 0, 2, 0] survivors: 4 of 8, each scaled by 1/(1-rate)
from statistics-ml
See also
sigmoid — documented in the same entry.