sarsa

Builtin index · Statistics & ML

result = q_learning(env, episodes, [alpha=0.1], [gamma=0.9], [epsilon=0.1], [seed=])

Input: env, episodes, alpha, gamma, epsilon, seed · Output: result

What it is

Tabular temporal-difference learning on a gridworld_env env for an integer number of episodes, with learning rate alpha, discount factor gamma, and epsilon-greedy exploration rate epsilon (all numbers), plus an optional integer seed=. q_learning is off-policy (bootstraps off the greedy next action); sarsa is on-policy (bootstraps off the action actually taken next). Both return a fitted model Record with .q_table (an (n_states, n_actions) matrix) and .rewards (a length-episodes vector, one total reward per episode — it should trend upward as training progresses), plus .predict(state) giving the greedy action (an integer) for a given state index.

Overloads

One call shape — this function is not overloaded.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

env = gridworld_env(4)
start = env.reset()
print("start state: {start}")

q = q_learning(env, 300, alpha = 0.3, gamma = 0.9, epsilon = 0.1, seed = 7)
s = sarsa(env, 300, alpha = 0.3, gamma = 0.9, epsilon = 0.1, seed = 7)
print("q_learning reward: early {round(mean(q.rewards[0:19]), 2)} -> late {round(mean(q.rewards[280:299]), 2)}")
print("sarsa reward:      early {round(mean(s.rewards[0:19]), 2)} -> late {round(mean(s.rewards[280:299]), 2)}")

state = env.reset()
steps = 0
done = false
while done == false and steps < 20
    action = q.predict(state)
    result = env.step(state, action)
    state = result.next_state
    steps = steps + 1
    done = result.done
end while
print("greedy policy reached the goal in {steps} steps")

start state: 0
q_learning reward: early -4 -> late 4.45
sarsa reward:      early -1.5 -> late 4.6
greedy policy reached the goal in 6 steps

from statistics-ml

How it works

q_learning(env, episodes, [alpha=0.1], [gamma=0.9], [epsilon=0.1], [max_steps=], seed=) — off-policy tabular Q-learning (Watkins 1989): the behavior policy is epsilon-greedy, but the update always bootstraps off the GREEDY next action regardless of which action is actually taken next. sarsa(...) (same signature) is the on-policy sibling: the update bootstraps off the SAME action epsilon-greedy actually picked for the next step (so that action, not a resampled one, is what the loop continues with). Both return a Value::Model (kind matches the builtin's own name) with .q_table (an (n_states, n_actions) Mat) and .rewards (a Vec, one total-episode-reward entry per episode — the real correctness signal for a working RL implementation: it should trend upward across training, not just "run without crashing").

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.

See also

q_learning — documented in the same entry.