lstm_init

Builtin index · Statistics & ML

result = lstm_init(input_size, hidden_size, [seed=])

Input: input_size, hidden_size, seed · Output: result

What it is

Builds fresh LSTM weights for an input of size input_size (integer, number of features per timestep) and a hidden state of size hidden_size (integer): Xavier-scaled weight matrices and zero biases for all four gates (forget, input, output, candidate), plus an optional integer seed=.

Overloads

One call shape — this function is not overloaded.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

raw = lstm_init(2, 3, seed = 1)
w = {wf=param(raw.wf), uf=param(raw.uf), bf=param(raw.bf),
     wi=param(raw.wi), ui=param(raw.ui), bi=param(raw.bi),
     wo=param(raw.wo), uo=param(raw.uo), bo=param(raw.bo),
     wg=param(raw.wg), ug=param(raw.ug), bg=param(raw.bg)}
Xseq = [0.5, -0.3, 0.8; 0.2, 0.4, -0.1]   # (input_size, seq_len)

hs = lstm_forward(Xseq, [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], w)
h_last = hs[length(hs) - 1]
loss = sum(h_last .^ 2)
g = grad(loss, wrt = w.wf)                # through time, on the same tape

print("loss: {round(loss, 4)}")
print("dL/dwf: {round(g, 4)}")

loss: 0.0027
dL/dwf: [0.0001, 0; 0.0005, 0; 0.0021, -0.0002]

from statistics-ml

How it works

lstm_init(input_size, hidden_size, [seed=]) — weight initializer, the LSTM sibling of gru_init: same seed= reproducibility convention, same Xavier/Glorot scaling (std = 1/sqrt(fan_in), fan-in input_size for the Ws and hidden_size for the Us), same zero-initialized biases and same "returns plain untracked values, wrap the ones you want trained in param(...) yourself" contract. Twelve fields instead of GRU's nine — four gates (f/i/o/g) instead of three (z/r/h) — wf/uf/bf, wi/ui/bi, wo/uo/bo, wg/ug/bg.

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.