softmax_rows

Builtin index · Statistics & ML

result = softmax_rows(M)

Input: M · Output: result

What it is

Softmax along each row of an N-row, D-column matrix M of scores, shifted by the row maximum first so a large score cannot overflow the exponential.

Overloads

One call shape — this function is not overloaded.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

w = param([0.5, -0.3; 0.2, 0.8])
b = param([0.1, -0.1])
x = [1.0, 2.0; -0.5, 1.5]      # two examples, two features each

z = dense(x, w, b)
a = relu(z)
p = sigmoid(a)
loss = sum(p .^ 2)
g = grad(loss, wrt = w)
print("relu:    {round(a, 4)}")
print("sigmoid: {round(p, 4)}")
print("dL/dw:   {round(g, 4)}")
tape_reset()                    # clear the tape before the next computation


scores = [2.0, 1.0, 0.1; 0.5, 0.5, 0.5]
probs = softmax_rows(scores)
print("softmax row 0: {round(probs[0, :], 4)}")

seed(3)
kept = dropout(ones(1, 8), 0.5)
print("dropout output: {kept}")
print("survivors:      {sum(kept > 0)} of 8, each scaled by 1/(1-rate)")

relu:    [1, 1.4; 0, 1.25]
sigmoid: [0.7311, 0.8022; 0.5, 0.7773]
dL/dw:   [0.2875, 0.12; 0.5749, 0.9128]
softmax row 0: [0.659, 0.2424, 0.0986]
dropout output: [2, 2, 0, 0, 2, 0, 2, 0]
survivors:      4 of 8, each scaled by 1/(1-rate)

from statistics-ml

How it works

softmax_rows(x) — row-wise softmax, exposed as its own builtin (not just an internal helper of attention) so it can be gradient-checked in isolation from the surrounding matmuls — softmax's backward is the single most bug-prone step in the whole attention block.

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.