tape_reset

Builtin index · Statistics & ML

tape_reset()

Input: no input · Output: no output

What it is

Discards every entry recorded so far on the autodiff tape, freeing the memory a chain of param/grad calls has built up. Takes no arguments; call it between independent computations, since a program that never calls it keeps every tracked operation alive for as long as it runs.

Overloads

One call shape — this function is not overloaded.

Code	Input	Output	Description
`tape_reset()`	no input	no output	Discards every entry recorded so far on the autodiff tape, freeing the memory a chain of `param`/`grad` calls has built up. Takes no arguments; call it between independent computations, since a program that never calls it keeps every tracked operation alive for as long as it runs.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

raw = gru_init(2, 3, seed = 1)
w = {wz=param(raw.wz), uz=param(raw.uz), bz=param(raw.bz),
     wr=param(raw.wr), ur=param(raw.ur), br=param(raw.br),
     wh=param(raw.wh), uh=param(raw.uh), bh=param(raw.bh)}
Xseq = [0.5, -0.3, 0.8; 0.2, 0.4, -0.1]   # (input_size, seq_len)

hs = gru_forward(Xseq, [0.0, 0.0, 0.0], w)
h_last = hs[length(hs) - 1]
print("gru_forward last hidden state: {round(h_last, 4)}")

loss = sum(h_last .^ 2)
g = grad(loss, wrt = w.wz)
print("dL/dwz: {round(g, 4)}")
tape_reset()

h1 = gru_cell([1.0], [1.0], {wz=0.5, uz=0.5, bz=0.0, wr=0.3, ur=0.3, br=0.0, wh=0.2, uh=0.2, bh=0.0})
print("gru_cell one step: {round(h1, 4)}")

output

gru_forward last hidden state: [-0.542; -0.0036; -0.0815]
dL/dwz: [0.2321, -0.0486; -0.0006, 0.0001; -0.0079, 0.0034]
gru_cell one step: [0.5012]

from statistics-ml

w = param([0.5, -0.3; 0.2, 0.8])
b = param([0.1, -0.1])
x = [1.0, 2.0; -0.5, 1.5]      # two examples, two features each

z = dense(x, w, b)
a = relu(z)
p = sigmoid(a)
loss = sum(p .^ 2)
g = grad(loss, wrt = w)
print("relu:    {round(a, 4)}")
print("sigmoid: {round(p, 4)}")
print("dL/dw:   {round(g, 4)}")
tape_reset()                    # clear the tape before the next computation


scores = [2.0, 1.0, 0.1; 0.5, 0.5, 0.5]
probs = softmax_rows(scores)
print("softmax row 0: {round(probs[0, :], 4)}")

seed(3)
kept = dropout(ones(1, 8), 0.5)
print("dropout output: {kept}")
print("survivors:      {sum(kept > 0)} of 8, each scaled by 1/(1-rate)")

output

relu:    [1, 1.4; 0, 1.25]
sigmoid: [0.7311, 0.8022; 0.5, 0.7773]
dL/dw:   [0.2875, 0.12; 0.5749, 0.9128]
softmax row 0: [0.659, 0.2424, 0.0986]
dropout output: [2, 2, 0, 0, 2, 0, 2, 0]
survivors:      4 of 8, each scaled by 1/(1-rate)

from statistics-ml

x = [1.0, 2.0, 3.0, 4.0]
y_true = [2.0, 4.0, 6.0, 8.0]
w = param(0.0)
function loss_fn(w)
    pred = w * x
    diff = pred - y_true
    return mean(diff .^ 2)
end function
opt = sgd(w, 0.01)
losses = train_loop("loss_fn", w, 100, optimizer = "opt")
print("train_loop (sgd): loss {round(losses[0], 3)} -> {round(losses[99], 5)}")
tape_reset()

w2v = 0.0
opt2 = adam(param(w2v), lr = 0.1)
for t = 0 to 19
    tape_reset()
    w2 = param(w2v)
    pred = w2 * x
    loss2 = mean((pred - y_true) .^ 2)
    g = grad(loss2, wrt = w2)
    opt2 = optimizer_step(opt2, g)
    w2v = opt2._params[0]
end for
print("optimizer_step (adam): w -> {round(w2v, 4)} (target 2.0)")
tape_reset()

# One step each from w=1.0, grad=3.0 -- same optimizer_step(opt, grad) call.

w3 = param(1.0)
grad3 = [3.0]
r1 = optimizer_step(nesterov_sgd(w3, lr = 0.05, momentum = 0.9), grad3)
r2 = optimizer_step(adamw(w3, lr = 0.05), grad3)
r3 = optimizer_step(nadam(w3, lr = 0.05), grad3)
r4 = optimizer_step(adamax(w3, lr = 0.05), grad3)
r5 = optimizer_step(rmsprop(w3, lr = 0.05), grad3)
r6 = optimizer_step(adagrad(w3, lr = 0.05), grad3)
r7 = optimizer_step(adadelta(w3), grad3)
print("nesterov_sgd -> {round(r1._params[0], 4)}, adamw -> {round(r2._params[0], 4)}, nadam -> {round(r3._params[0], 4)}")
print("adamax -> {round(r4._params[0], 4)}, rmsprop -> {round(r5._params[0], 4)}, adagrad -> {round(r6._params[0], 4)}, adadelta -> {round(r7._params[0], 4)}")

Xxor = [0.0, 0.0; 0.0, 1.0; 1.0, 0.0; 1.0, 1.0]
Yxor = [0.0; 1.0; 1.0; 2.0]
net = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
      compile(optimizer = lr_plateau(lr = 0.1, patience = 5, base = "adam"), loss = "mse", seed = 7)
net = net.fit(Xxor, Yxor, epochs = 60)
n = length(net.loss_history)
print("compile + fit(lr_plateau): loss {round(net.loss_history[0], 4)} -> {round(net.loss_history[n - 1], 4)}")

net2 = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
       compile(optimizer = lr_adaptive(lr = 0.05), loss = "mse", seed = 7)
net2 = net2.fit(Xxor, Yxor, epochs = 60)
n2 = length(net2.loss_history)
print("compile + fit(lr_adaptive): loss {round(net2.loss_history[0], 4)} -> {round(net2.loss_history[n2 - 1], 4)}")

quick = quick_mlp(Xxor, Yxor, seed = 3)
nq = length(quick.loss_history)
print("quick_mlp: loss {round(quick.loss_history[0], 4)} -> {round(quick.loss_history[nq - 1], 4)}")

output

train_loop (sgd): loss 30 -> 0
optimizer_step (adam): w -> 1.7775 (target 2.0)
nesterov_sgd -> 0.715, adamw -> 0.9495, nadam -> 0.9263
adamax -> 0.95, rmsprop -> 0.8419, adagrad -> 0.95, adadelta -> 0.9996
compile + fit(lr_plateau): loss 3.3387 -> 0.0341
compile + fit(lr_adaptive): loss 0.3897 -> 0.0002
quick_mlp: loss 3.1877 -> 0.0922

from statistics-ml

How it works

tape_reset() — frees every recorded TapeEntry (memory hygiene for a long-running script that keeps building new Tensors and never needs to grad() against an old one). Danger: any Value::Tensor created *before* this call has its .node index invalidated — the tape's own numbering starts over from 0, so a stale Tensor's old index can silently collide with a brand-new node created after the reset, corrupting *that* node's gradient rather than erroring (traced down inside train_loop, which used to clear the tape and then keep reusing the same pre-reset Tensor every epoch — fixed there by building a genuinely fresh leaf Tensor each time, right after each reset, never carrying one across the boundary). Safe usage: reset only when nothing you still hold a Value::Tensor reference to will be used in a later grad() call — re-wrap any value you need to keep training through via param()/track() again after resetting.

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.