lr_plateau

Builtin index · Statistics & ML

result = lr_plateau([lr=], [patience=], [factor=], [base=])

Input: lr, patience, factor, base · Output: result

What it is

Learning-rate controllers that wrap a base optimizer rather than replacing it: lr= the starting learning rate (number), patience an integer number of epochs to wait for improvement before shrinking the rate, factor the multiplicative shrink (a number in (0, 1)), and base a string naming the wrapped optimizer (e.g. "adam"). Passed as optimizer= wherever a plain optimizer recipe would go.

Overloads

One call shape — this function is not overloaded.

Code	Input	Output	Description
`result = lr_plateau([lr=], [patience=], [factor=], [base=])`	`lr` number or string, optionalnot documented yet `patience` number or string, optionalnot documented yet `factor` number or string, optionalnot documented yet `base` number or string, optionalnot documented yet	`result` Record	Returns an optimizer-recipe Record.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

x = [1.0, 2.0, 3.0, 4.0]
y_true = [2.0, 4.0, 6.0, 8.0]
w = param(0.0)
function loss_fn(w)
    pred = w * x
    diff = pred - y_true
    return mean(diff .^ 2)
end function
opt = sgd(w, 0.01)
losses = train_loop("loss_fn", w, 100, optimizer = "opt")
print("train_loop (sgd): loss {round(losses[0], 3)} -> {round(losses[99], 5)}")
tape_reset()

w2v = 0.0
opt2 = adam(param(w2v), lr = 0.1)
for t = 0 to 19
    tape_reset()
    w2 = param(w2v)
    pred = w2 * x
    loss2 = mean((pred - y_true) .^ 2)
    g = grad(loss2, wrt = w2)
    opt2 = optimizer_step(opt2, g)
    w2v = opt2._params[0]
end for
print("optimizer_step (adam): w -> {round(w2v, 4)} (target 2.0)")
tape_reset()

# One step each from w=1.0, grad=3.0 -- same optimizer_step(opt, grad) call.

w3 = param(1.0)
grad3 = [3.0]
r1 = optimizer_step(nesterov_sgd(w3, lr = 0.05, momentum = 0.9), grad3)
r2 = optimizer_step(adamw(w3, lr = 0.05), grad3)
r3 = optimizer_step(nadam(w3, lr = 0.05), grad3)
r4 = optimizer_step(adamax(w3, lr = 0.05), grad3)
r5 = optimizer_step(rmsprop(w3, lr = 0.05), grad3)
r6 = optimizer_step(adagrad(w3, lr = 0.05), grad3)
r7 = optimizer_step(adadelta(w3), grad3)
print("nesterov_sgd -> {round(r1._params[0], 4)}, adamw -> {round(r2._params[0], 4)}, nadam -> {round(r3._params[0], 4)}")
print("adamax -> {round(r4._params[0], 4)}, rmsprop -> {round(r5._params[0], 4)}, adagrad -> {round(r6._params[0], 4)}, adadelta -> {round(r7._params[0], 4)}")

Xxor = [0.0, 0.0; 0.0, 1.0; 1.0, 0.0; 1.0, 1.0]
Yxor = [0.0; 1.0; 1.0; 2.0]
net = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
      compile(optimizer = lr_plateau(lr = 0.1, patience = 5, base = "adam"), loss = "mse", seed = 7)
net = net.fit(Xxor, Yxor, epochs = 60)
n = length(net.loss_history)
print("compile + fit(lr_plateau): loss {round(net.loss_history[0], 4)} -> {round(net.loss_history[n - 1], 4)}")

net2 = input(2) |> dense(4, activation = "relu") |> dense(1, activation = "none") |>
       compile(optimizer = lr_adaptive(lr = 0.05), loss = "mse", seed = 7)
net2 = net2.fit(Xxor, Yxor, epochs = 60)
n2 = length(net2.loss_history)
print("compile + fit(lr_adaptive): loss {round(net2.loss_history[0], 4)} -> {round(net2.loss_history[n2 - 1], 4)}")

quick = quick_mlp(Xxor, Yxor, seed = 3)
nq = length(quick.loss_history)
print("quick_mlp: loss {round(quick.loss_history[0], 4)} -> {round(quick.loss_history[nq - 1], 4)}")

output

train_loop (sgd): loss 30 -> 0
optimizer_step (adam): w -> 1.7775 (target 2.0)
nesterov_sgd -> 0.715, adamw -> 0.9495, nadam -> 0.9263
adamax -> 0.95, rmsprop -> 0.8419, adagrad -> 0.95, adadelta -> 0.9996
compile + fit(lr_plateau): loss 3.3387 -> 0.0341
compile + fit(lr_adaptive): loss 0.3897 -> 0.0002
quick_mlp: loss 3.1877 -> 0.0922

from statistics-ml

lr_plateau

What it is

Overloads

Examples

See also