layer_norm

Builtin index · Statistics & ML

result = layer_norm(x, gamma, beta, [eps=])

Input: x, gamma, beta, eps · Output: result

What it is

Normalizes each row of an N-row, D-column matrix x independently to zero mean and unit variance, then applies a learned length-D per-feature scale gamma and shift beta, with eps (a small positive number, default a tiny constant) added inside the variance's square root for numerical stability.

Overloads

One call shape — this function is not overloaded.

Code	Input	Output	Description
`result = layer_norm(x, gamma, beta, [eps=])`	`x` matrix or numbernot documented yet `gamma` matrix or numbernot documented yet `beta` matrix or numbernot documented yet `eps` matrix or number, optionalnot documented yet	`result` matrix	Returns an `(N, D)` matrix, the same shape as `x`.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

x = [1.0, 2.0, 3.0; 10.0, 20.0, 30.0]
normed = layer_norm(x, [1.0, 1.0, 1.0], [0.0, 0.0, 0.0])
print("layer_norm (scale-invariant per row): {normed}")

q = [1.0, 0.5; -0.5, 1.0; 0.2, -0.3]
k = [1.0, 0.0; 0.0, 1.0; 0.5, 0.5]
v = [1.0, 2.0; 3.0, 4.0; 5.0, 6.0]
attn = scaled_dot_product_attention(q, k, v)
print("attention output: {round(attn, 3)}")

eye2 = [1.0, 0.0; 0.0, 1.0]
head = {wq = eye2, wk = eye2, wv = eye2, wo = eye2}
mh = multi_head_attention(q, (head,), 1)
plain = scaled_dot_product_attention(q, q, q)
print("multi_head_attention with 1 identity head matches plain self-attention: {sum(abs(mh - plain)) < 1e-9}")

weights = {heads = (head,), ln1_gamma = [1.0, 1.0], ln1_beta = [0.0, 0.0],
           w1 = [0.2, -0.1; 0.1, 0.3], b1 = [0.0, 0.0],
           w2 = [0.1, -0.2; 0.3, 0.1], b2 = [0.0, 0.0],
           ln2_gamma = [1.0, 1.0], ln2_beta = [0.0, 0.0]}
block_out = transformer_block(q, weights)
print("transformer_block: {round(block_out, 3)}")

output

layer_norm (scale-invariant per row): [-1.224736, 0, 1.224736; -1.224745, 0, 1.224745]
attention output: [2.872, 3.872; 3.25, 4.25; 2.872, 3.872]
multi_head_attention with 1 identity head matches plain self-attention: true
transformer_block: [1, -1; -1, 1; 1, -1]

from statistics-ml

How it works

layer_norm(x, gamma, beta, [eps=1e-5]) — normalizes each ROW of x (one row = one token/sample's feature vector, (n, d) overall) to zero mean / unit variance, then rescales with a learned per-feature gamma and shifts by beta (both length-d, broadcasting as a (1, d) row against x exactly the way dense's own X*W + b bias already broadcasts). Composed ENTIRELY from already-tracked primitives (tb's "-"/".*"/"+"/".*", trr's new "row_mean", tm1's "sqrt") — no bespoke tensor_vjp rule for layer_norm itself, only for the two small row-reduction primitives it (and softmax) both build on. Layer norm's mean/variance backward terms are a well-known place autodiff implementations get subtly wrong, so this was gradient- checked against finite differences before shipping (see the acceptance tests) rather than just visually reasoned through.

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.