dropout

Builtin index · Statistics & ML

result = dropout(x, rate)

Input: x, rate · Output: result

What it is

Zeroes a random share of the entries of x (a vector or matrix of any shape) at probability rate (a number in [0, 1)), and scales the surviving entries up by 1 / (1 - rate), so the expected value of the output equals x.

Overloads

One call shape — this function is not overloaded.

Code	Input	Output	Description
`result = dropout(x, rate)`	`x` vector, matrix, or numbernot documented yet `rate` vector, matrix, or numbernot documented yet	`result` —	Returns a value the same shape as `x`, freshly randomized on every call.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

w = param([0.5, -0.3; 0.2, 0.8])
b = param([0.1, -0.1])
x = [1.0, 2.0; -0.5, 1.5]      # two examples, two features each

z = dense(x, w, b)
a = relu(z)
p = sigmoid(a)
loss = sum(p .^ 2)
g = grad(loss, wrt = w)
print("relu:    {round(a, 4)}")
print("sigmoid: {round(p, 4)}")
print("dL/dw:   {round(g, 4)}")
tape_reset()                    # clear the tape before the next computation


scores = [2.0, 1.0, 0.1; 0.5, 0.5, 0.5]
probs = softmax_rows(scores)
print("softmax row 0: {round(probs[0, :], 4)}")

seed(3)
kept = dropout(ones(1, 8), 0.5)
print("dropout output: {kept}")
print("survivors:      {sum(kept > 0)} of 8, each scaled by 1/(1-rate)")

output

relu:    [1, 1.4; 0, 1.25]
sigmoid: [0.7311, 0.8022; 0.5, 0.7773]
dL/dw:   [0.2875, 0.12; 0.5749, 0.9128]
softmax row 0: [0.659, 0.2424, 0.0986]
dropout output: [2, 2, 0, 0, 2, 0, 2, 0]
survivors:      4 of 8, each scaled by 1/(1-rate)

from statistics-ml