1. What is Supervised Learning?

Traditionally, a computational problem is specified as a mapping from inputs to outputs. A human programmer designs an algorithm that explicitly describes how each input should be transformed into the desired output.

In supervised learning, instead of manually specifying all rules, a learning algorithm is given a collection of examples and automatically discovers the relationship between inputs and outputs.

• Feature: A measurable characteristic or attribute of an instance.
• Label / Target: The value the model aims to predict.
• Training Set: Data used to learn the model.
• Test Set: Data used to evaluate generalization.
• Model: The learned predictive function.

• Classification: Predict a discrete category, such as spam vs. non-spam email.
• Regression: Predict a continuous numerical value, such as house price from size, age, location.

2. Learning Decision Trees

A Decision Tree Learner algorithm builds a decision tree from the top down by repeatedly divide the sample data, as in Example 7.11.

3. Linear regression

• x = input feature
• w = learned weight
• b = bias term

Binary classification uses a squashed linear function to separate positive and negative instances of a concept by applying a non-linear activation function to a linear combination of inputs, to "squash" the output into the range of [0, 1].

A Boolean classification is linearly separable if there exists a hyperplane where the classification is true on one side of the hyperplane and false on the other side.

Gradient Descent is an iterative method to find a local minimum of a function. Stochastic gradient descent updates the weights using one random sample at a time.

When the domain of the target variable is categorical with more than two values, indicator variables can be used to convert the classification to binary variables.

4. Composite Models

Ockham's Razor states that, among competing hypotheses that explain the data equally well, the simplest one should generally be preferred. It is often used to reduce overfitting.

A kernel function implicitly maps data into a higher-dimensional space with new features to turn previously difficult classification problems into linearly separable, as in Example 7.20.

Ensemble methods combine multiple learners to obtain stronger performance than individual models. Boosting combines multiple weak learners, each of which learns from the errors of the previous ones, to produce a stronger predictive model

5. Learning belief networks

• When the network structure is known, conditional probabilities can be learned from labeled observations using supervised learning.
• A hidden variable or a latent variable is a variable in a belief network whose value is not observed for any of the examples, as shown in Figure 10.9. The expectation maximization (EM) algorithm for learning belief networks with hidden variables will be introduced in the lecture on Unsupervised Learning.
• Handling missing data requires a causal model of how the data is missing, as described in Section 11.2.
• The network structure can be learned from a total ordering of variables, and the parents of each variable in it determined by conditional independence assumptions.
• Another way to learn the structure is the search and score method (10.4.3).