3203. Introduction to Artificial Intelligence
Markov Models
1. Markov Chains
A Markov Chain is a belief network with random variables in a sequence, where each variable only directly depends on its predecessor in the sequence, which is called the Markov Property.
A Hidden Markov Model (HMM) augments a Markov chain with an observation process, allowing the underlying states to remain hidden while producing observable outputs.
A Naive Bayes classifier is a belief network that has a single node – the class – that directly influences the other variables, and the other variables are independent of each other given the class.
2. Sequential Decisions
A sequential decision problem describes situations where an agent must make a sequence of decisions over time. It is an extension of single-step decision making.
A model must specify:
- What actions are available at each stage.
- What information is available when decisions are made.
- The consequences of actions.
- How desirable the outcomes are.
Goal: Choosing actions that maximize the expected discounted sum of rewards (utility).
A decision network is a directed acyclic graph (DAG) with chance nodes (drawn as ovals), decision nodes (drawn as rectangles), and a utility node (drawn as a diamond). A possible world corresponds to a complete history and specifies the values of all random and decision variables, including all observed variables.
A policy specifies what the agent will do for each of the possible observations. Each policy has an expected utility, obtained by averaging the utility over the worlds that satisfy this policy.
3. Markov Decision Process
A Markov Decision Process (MDP) specifies:
- States
- Actions
- Transition probabilities
- Rewards
Rewards provide a way to factor utility through time, by having a reward for each time, and accumulating the rewards to determine utility. A discount factor in [0, 1) is multiplied to a (one-step) future reward to get an equivalent current reward.
The expected value of a policy is given by Bellman expectation equation (12.2), i.e., "value = immediate reward + expected future utility", which can be calculated using two iterative algorithms:
-
Value Iteration can be viewed as propagating utility information backward through the state space. However, because the state-transition graph generally contains cycles, there is no true endpoint from which all values can be computed directly. Therefore, the algorithm repeatedly refines state values (estimated utilities) until they converge.
-
Policy Iteration can be viewed as refining decision-making through successive approximations. The algorithm alternates between policy evaluation, which estimates the utility of states under the current policy, and policy improvement, which updates the policy based on those estimates, until the policy converges to an optimal one.