dense

Builtin index · Statistics & ML

result = dense(x, w, b)

Input: x, w, b · Output: result

What it is

One fully-connected layer on an N-row, D-column input matrix x, a (D, H) weight matrix w, and a length-H bias vector b: computes x * w + b, broadcasting b over every row.

Overloads

One call shape — this function is not overloaded.

Code	Input	Output	Description
`result = dense(x, w, b)`	`x` matrix or vectornot documented yet `w` matrix or vectornot documented yet `b` matrix or vectornot documented yet	`result` matrix	Returns an `(N, H)` matrix.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

w = param([0.5, -0.3; 0.2, 0.8])
b = param([0.1, -0.1])
x = [1.0, 2.0; -0.5, 1.5]      # two examples, two features each

z = dense(x, w, b)
a = relu(z)
p = sigmoid(a)
loss = sum(p .^ 2)
g = grad(loss, wrt = w)
print("relu:    {round(a, 4)}")
print("sigmoid: {round(p, 4)}")
print("dL/dw:   {round(g, 4)}")
tape_reset()                    # clear the tape before the next computation


scores = [2.0, 1.0, 0.1; 0.5, 0.5, 0.5]
probs = softmax_rows(scores)
print("softmax row 0: {round(probs[0, :], 4)}")

seed(3)
kept = dropout(ones(1, 8), 0.5)
print("dropout output: {kept}")
print("survivors:      {sum(kept > 0)} of 8, each scaled by 1/(1-rate)")

output

relu:    [1, 1.4; 0, 1.25]
sigmoid: [0.7311, 0.8022; 0.5, 0.7773]
dL/dw:   [0.2875, 0.12; 0.5749, 0.9128]
softmax row 0: [0.659, 0.2424, 0.0986]
dropout output: [2, 2, 0, 0, 2, 0, 2, 0]
survivors:      4 of 8, each scaled by 1/(1-rate)

from statistics-ml