gridworld_env

Builtin index · Statistics & ML

result = gridworld_env(n, [start=], [goal=], [step_reward=-1], [goal_reward=10])

Input: n, start, goal, step_reward, goal_reward · Output: result

What it is

Builds an n x n grid environment (integer n), with start/goal (optional integer state indices, defaulting to opposite corners), step_reward (number, default -1, given on every non-terminal move) and goal_reward (number, default 10, given on reaching the goal). Actions are the integers 0 to 3: up, right, down, left; walking into an edge leaves the agent where it was.

Overloads

One call shape — this function is not overloaded.

Examples

From the documentation, and run by the build -- the output and the figure below are what the engine actually produced. Simplest first.

env = gridworld_env(4)
start = env.reset()
print("start state: {start}")

q = q_learning(env, 300, alpha = 0.3, gamma = 0.9, epsilon = 0.1, seed = 7)
s = sarsa(env, 300, alpha = 0.3, gamma = 0.9, epsilon = 0.1, seed = 7)
print("q_learning reward: early {round(mean(q.rewards[0:19]), 2)} -> late {round(mean(q.rewards[280:299]), 2)}")
print("sarsa reward:      early {round(mean(s.rewards[0:19]), 2)} -> late {round(mean(s.rewards[280:299]), 2)}")

state = env.reset()
steps = 0
done = false
while done == false and steps < 20
    action = q.predict(state)
    result = env.step(state, action)
    state = result.next_state
    steps = steps + 1
    done = result.done
end while
print("greedy policy reached the goal in {steps} steps")

start state: 0
q_learning reward: early -4 -> late 4.45
sarsa reward:      early -1.5 -> late 4.6
greedy policy reached the goal in 6 steps

from statistics-ml

How it works

Q-learning/SARSA (2026-08-25) ---- Follows the SAME Value::Model protocol every other ML estimator in this codebase already uses (see model::ModelHandle's own doc comment) — no new grammar, env.reset()/env.step(...) reach these through the existing recv.method(args) sugar (eval_call). One deliberate deviation from the textbook Gym-style next_state, reward, done = env.step(action) (env remembers its own current position): every other stateful-looking Model in this codebase (the kalman/particle_filter estimation family's state.update(...)) is immutable and returns a brand-new state rather than mutating itself in place — there is no interior-mutability convention for Model anywhere else to reuse, and inventing one just for this feature would be a bigger, riskier change than tabular RL itself. So gridworld_env(...) is a pure, immutable description of the MDP (grid size, start, goal, rewards) with no live internal position, and step takes the current state explicitly — env.step(state, action) — exactly the same shape as state = state.update(...) for the estimation-theory family. q_learning/sarsa below track "current state" as a plain number in their own training loop, the same way a script driving one episode by hand would. gridworld_env(n, [start=], [goal=], [step_reward=], [goal_reward=]) — the one concrete environment this pass validates the interface against: an n x n grid, states indexed row-major 0..n*n-1, agent starts at start (default 0, the top-left corner) and must reach goal (default n*n-1, the bottom-right corner); a small negative step_reward (default -1) every move, a large positive goal_reward (default 10) on arrival. Actions are 0..3 (up/right/down/left — see gridworld_transition); moving off the grid edge is a no-op (the agent stays put), not an error and not a wraparound. Deliberately no walls in this first silently dropped).

From the note above this builtin in the interpreter, where the reasoning was written down as it was built.