3203. Introduction to Artificial Intelligence
Reinforcement Learning
1. The problem
In reinforcement learning, an agent learns how to behave through interaction with an environment. At each step, the agent observes its current state, selects an action, and receives a reward. The objective is to learn a policy that maximizes the expected long-term cumulative reward. This is often formulated as a Markov Decision Process (MDP), especially when the transition function P(s'|s,a) and reward function R(s,a) are initially unknown. In such cases, "model-free" learning methods are needed.
Example 13.1: a simple MDP
Conceptual issues:
2. Q-learning
In reinforcement learning, an agent tries to learn the optimal policy from its history of interaction with the environment. A history of an agent is a sequence of "experiences", each typically represented as (s, a, r, s'), where s is the current state, a is the selected action, r is the reward received, and s' is the resulting state.
Q-learning approximates value iteration by using temporal difference updates based on sampled experiences to learn the optimal action-value function and, consequently, the optimal policy:
Q(s,a) := Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]
where
α = learning rate,
γ = discount factor,
r = immediate reward,
s' = next state
The agent maintains a table of Q[S,A], where S is the set of states and A is the set of actions. Q[s,a] represents its current estimate of the optimal value.
An experience provides one data point for the value of Q[s,a], so it is used in updating.
Figure 13.3: Q-learning controller
When an action is selected in Q-learning, a balance between exploitation and exploration needs to be kept. A common strategy is e-greedy exploration. With probability e, the agent chooses a random action to explore the environment; otherwise, it selects the action with the highest estimated Q-value. As learning progresses, e is often gradually reduced so that exploration decreases and exploitation increases.
An on-policy learner learns the value of the policy being carried out by the agent, including the exploration steps.
An alternative to doing one Q-value update after each action is to use the experiences to learn a model, i.e., the P and R functions.
4. Applications
Demo: Reinforcement Learning Grid World