3203. Introduction to Artificial Intelligence
Markov Models
A Markov chain is a belief network with random variables in a sequence, where each variable only directly depends on its predecessor in the sequence, which is called the Markov Property.
A hidden Markov model (HMM) is an extension of a Markov chain by including observations.
A naive Bayes classifier is a belief network that has a single node – the class – that directly influences the other variables, and the other variables are independent of each other given the class.
2. Sequential Decisions
A sequential decision problem describes situations where an agent must make a sequence of decisions over time.
A model must specify:
- What actions are available at each stage.
- What information is available when decisions are made.
- The consequences of actions.
- How desirable the outcomes are.
Goal: Choosing actions that maximize the expected discounted sum of rewards (utility).
A decision network is a directed acyclic graph (DAG) with chance nodes (drawn as ovals), decision nodes (drawn as rectangles), and a utility node (drawn as a diamond).
A policy specifies what the agent will do for each of the possible observations.
A possible world corresponds to a complete history and specifies the values of all random and decision variables, including all observed variables.
Each policy has an expected utility, obtained by averaging the utility over the worlds that satisfy this policy.
A Markov Decision Process (MDP):
- States
- Actions
- Transition probabilities
- Rewards
Rewards provide a way to factor utility through time, by having a reward for each time, and accumulating the rewards to determine utility.
A discount factor in [0, 1) is multiplied to a (one-step) future reward to get an equivalent current reward.
The expected value of a policy is given by Bellman expectation equation (12.2), i.e., "value = immediate reward + expected future utility", which can be calculated using two iterative algorithms:
-
Value Iteration starts at the “end” and then works backward, repeatedly updates state values (estimated utility) until convergence:
- Initialize state values.
- Update values using expected future rewards.
- Repeat until values stabilize.
- Extract the optimal policy.
-
Policy Iteration starts with an arbitrary policy and iteratively improves it:
- Policy Evaluation: Determine the value of the current policy.
- Policy Improvement: Improve the policy using the computed values.