adadelta
Builtin index · Statistics & ML
result = rmsprop(params, [lr=0.001])
Input: params, lr · Output: result
What it is
Per-parameter learning rates derived from the running gradient history, given params (a Tensor to optimize) and lr (a number, default 0.001). adagrad accumulates the full gradient history and so its effective rate only shrinks over time; rmsprop and adadelta use a decaying average instead, so they keep adapting late in training.
Overloads
One call shape — this function is not overloaded.
| Code | Input | Output | Description |
result = rmsprop(params, [lr=0.001]) | params numbernot documented yetlr number, optional (default 0.001)not documented yet | result Record | Returns an optimizer Record, or a recipe Record when called with no params. |
Examples
From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.
x = [1.0, 2.0, 3.0, 4.0]
y_true = [2.0, 4.0, 6.0, 8.0]
w = param(0.0)
function loss_fn(w)
pred = w * x
diff = pred - y_true
return mean(diff .^ 2)
end function
opt = sgd(w, 0.01)
losses = train_loop("loss_fn", w, 100, optimizer = "opt")
print("train_loop (sgd): loss {round(losses[0], 3)} -> {round(losses[99], 5)}")
tape_reset()
w2v = 0.0
opt2 = adam(param(w2v), lr = 0.1)
for t = 0 to 19
tape_reset()
w2 = param(w2v)
pred = w2 * x
loss2 = mean((pred - y_true) .^ 2)
g = grad(loss2, wrt = w2)
opt2 = optimizer_step(opt2, g)
w2v = opt2._params[0]
end for
print("optimizer_step (adam): w -> {round(w2v, 4)} (target 2.0)")
tape_reset()
# One step each from w=1.0, grad=3.0 -- same optimizer_step(opt, grad) call.
w3 = param(1.0)
grad3 = [3.0]
r1 = optimizer_step(nesterov_sgd(w3, lr = 0.05, momentum = 0.9), grad3)
r2 = optimizer_step(adamw(w3, lr = 0.05), grad3)
r3 = optimizer_step(nadam(w3, lr = 0.05), grad3)
r4 = optimizer_step(adamax(w3, lr = 0.05), grad3)
r5 = optimizer_step(rmsprop(w3, lr = 0.05), grad3)
r6 = optimizer_step(adagrad(w3, lr = 0.05), grad3)
r7 = optimizer_step(adadelta(w3), grad3)
print("nesterov_sgd -> {round(r1._params[0], 4)}, adamw -> {round(r2._params[0], 4)}, nadam -> {round(r3._params[0], 4)}")
print("adamax -> {round(r4._params[0], 4)}, rmsprop -> {round(r5._params[0], 4)}, adagrad -> {round(r6._params[0], 4)}, adadelta -> {round(r7._params[0], 4)}")
Xxor = [0.0, 0.0; 0.0, 1.0; 1.0, 0.0; 1.0, 1.0]
Yxor = [0.0; 1.0; 1.0; 2.0]
net = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
compile(optimizer = lr_plateau(lr = 0.1, patience = 5, base = "adam"), loss = "mse", seed = 7)
net = net.fit(Xxor, Yxor, epochs = 60)
n = length(net.loss_history)
print("compile + fit(lr_plateau): loss {round(net.loss_history[0], 4)} -> {round(net.loss_history[n - 1], 4)}")
net2 = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
compile(optimizer = lr_adaptive(lr = 0.05), loss = "mse", seed = 7)
net2 = net2.fit(Xxor, Yxor, epochs = 60)
n2 = length(net2.loss_history)
print("compile + fit(lr_adaptive): loss {round(net2.loss_history[0], 4)} -> {round(net2.loss_history[n2 - 1], 4)}")
quick = quick_mlp(Xxor, Yxor, seed = 3)
nq = length(quick.loss_history)
print("quick_mlp: loss {round(quick.loss_history[0], 4)} -> {round(quick.loss_history[nq - 1], 4)}")
output
train_loop (sgd): loss 30 -> 0 optimizer_step (adam): w -> 1.7775 (target 2.0) nesterov_sgd -> 0.715, adamw -> 0.9495, nadam -> 0.9263 adamax -> 0.95, rmsprop -> 0.8419, adagrad -> 0.95, adadelta -> 0.9996 compile + fit(lr_plateau): loss 3.3387 -> 0.0341 compile + fit(lr_adaptive): loss 0.3897 -> 0.0002 quick_mlp: loss 3.1877 -> 0.0922
from statistics-ml