adamax
Builtin index · Statistics & ML
result = adam(params, [lr=0.001])
Input: params, lr · Output: result
What it is
The Adam family of per-parameter adaptive-learning-rate optimizers, given params (a Tensor to optimize) and a learning rate lr (a number, default 0.001). adamw decouples weight decay from the gradient update; nadam folds in Nesterov-style look-ahead momentum; adamax swaps Adam's second-moment estimate for an infinity norm.
Overloads
One call shape — this function is not overloaded.
| Code | Input | Output | Description |
result = adam(params, [lr=0.001]) | params numbernot documented yetlr number, optional (default 0.001)not documented yet | result Record | Returns an optimizer Record, or (with no params) a recipe Record for optimizer=. |
Examples
From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.
x = [1.0, 2.0, 3.0, 4.0]
y_true = [2.0, 4.0, 6.0, 8.0]
w = param(0.0)
function loss_fn(w)
pred = w * x
diff = pred - y_true
return mean(diff .^ 2)
end function
opt = sgd(w, 0.01)
losses = train_loop("loss_fn", w, 100, optimizer = "opt")
print("train_loop (sgd): loss {round(losses[0], 3)} -> {round(losses[99], 5)}")
tape_reset()
w2v = 0.0
opt2 = adam(param(w2v), lr = 0.1)
for t = 0 to 19
tape_reset()
w2 = param(w2v)
pred = w2 * x
loss2 = mean((pred - y_true) .^ 2)
g = grad(loss2, wrt = w2)
opt2 = optimizer_step(opt2, g)
w2v = opt2._params[0]
end for
print("optimizer_step (adam): w -> {round(w2v, 4)} (target 2.0)")
tape_reset()
# One step each from w=1.0, grad=3.0 -- same optimizer_step(opt, grad) call.
w3 = param(1.0)
grad3 = [3.0]
r1 = optimizer_step(nesterov_sgd(w3, lr = 0.05, momentum = 0.9), grad3)
r2 = optimizer_step(adamw(w3, lr = 0.05), grad3)
r3 = optimizer_step(nadam(w3, lr = 0.05), grad3)
r4 = optimizer_step(adamax(w3, lr = 0.05), grad3)
r5 = optimizer_step(rmsprop(w3, lr = 0.05), grad3)
r6 = optimizer_step(adagrad(w3, lr = 0.05), grad3)
r7 = optimizer_step(adadelta(w3), grad3)
print("nesterov_sgd -> {round(r1._params[0], 4)}, adamw -> {round(r2._params[0], 4)}, nadam -> {round(r3._params[0], 4)}")
print("adamax -> {round(r4._params[0], 4)}, rmsprop -> {round(r5._params[0], 4)}, adagrad -> {round(r6._params[0], 4)}, adadelta -> {round(r7._params[0], 4)}")
Xxor = [0.0, 0.0; 0.0, 1.0; 1.0, 0.0; 1.0, 1.0]
Yxor = [0.0; 1.0; 1.0; 2.0]
net = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
compile(optimizer = lr_plateau(lr = 0.1, patience = 5, base = "adam"), loss = "mse", seed = 7)
net = net.fit(Xxor, Yxor, epochs = 60)
n = length(net.loss_history)
print("compile + fit(lr_plateau): loss {round(net.loss_history[0], 4)} -> {round(net.loss_history[n - 1], 4)}")
net2 = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
compile(optimizer = lr_adaptive(lr = 0.05), loss = "mse", seed = 7)
net2 = net2.fit(Xxor, Yxor, epochs = 60)
n2 = length(net2.loss_history)
print("compile + fit(lr_adaptive): loss {round(net2.loss_history[0], 4)} -> {round(net2.loss_history[n2 - 1], 4)}")
quick = quick_mlp(Xxor, Yxor, seed = 3)
nq = length(quick.loss_history)
print("quick_mlp: loss {round(quick.loss_history[0], 4)} -> {round(quick.loss_history[nq - 1], 4)}")
output
train_loop (sgd): loss 30 -> 0 optimizer_step (adam): w -> 1.7775 (target 2.0) nesterov_sgd -> 0.715, adamw -> 0.9495, nadam -> 0.9263 adamax -> 0.95, rmsprop -> 0.8419, adagrad -> 0.95, adadelta -> 0.9996 compile + fit(lr_plateau): loss 3.3387 -> 0.0341 compile + fit(lr_adaptive): loss 0.3897 -> 0.0002 quick_mlp: loss 3.1877 -> 0.0922
from statistics-ml