The Resourceful Rover
A 10×10 grid-world Mars colony where a Q-learning agent must harvest water ice under continuous resource depletion. The agent's state is deliberately projected down to (row, col) only — discarding water level and ice held — to study exactly what that simplification costs. Reproduces the original paper's training-curve result across 10 independent runs.
Q-learning agent reproduces the paper's reward curve: 10 runs × 5,000 episodes, 100-episode moving average climbs from a trough of −126.0 to a plateau of −106.4 (paper: ~−125 → ~−108). Try the same training loop live, in-browser.
Train the agent, then watch it survive
The full environment and Q-learning loop below run in your browser — no server, no API calls. Click Train Agent to execute 5,000 episodes in about a second.
- →ε-greedy exploration decaying 0.4 → 0.01 (×0.9995/episode) over 5,000 episodes
- →Reward shaping: +10/ice unit delivered, +2 empty return, −1 move, −5 obstacle, −100 dry colony
- →Reproduces the paper's training curve to within ~2 reward units: trough −126.0 vs paper's ~−125, plateau −106.4 vs paper's ~−108
- →State deliberately projected to (row, col) only — the paper's central finding: this position-only state is why reward plateaus below zero instead of converging near zero
How it works
10×10 Mars grid, depleting resources
The rover navigates a discretized 10×10 grid representing a Mars colony. Water reserves decay every step; ice patches replenish supply. Episode ends when reserves hit zero, or at a 150-step cap.