CIS 5603. Artificial Intelligence

Learning as Inference

1. Non-Deductive Infernece

Traditional study on inference has been focused on deduction, which follows inference rules that are "truth-preserving" in the model-theoretic sense.

Non-deductive inference has been studied by many thinkers:

Aristotle considered induction as "argument from the particular to the universal", so is the opposite of deduction.
Bacon's method of induction requires systematic observations and generalizations.
Hume on induction: induction (and causal inference) presupposes that the future will resemble the past, so cannot be logically justified.
Peirce defined induction and abduction as "reversed deduction" in term logic. He also identified their cognitive functions as "generalization" and "explanation", respectively.

Since non-deductive inference (e.g., induction, abduction, and analogy) is generally "ampliative", in that it generates conclusions that go beyond the available premises, it can be viewed as a learning process.

2. Learning via non-deductive inference

In AI research, various attempts have been made to use non-deductive inference in learning. Unlike functional and algorithmic approaches, inferential approaches focus on the justification of individual steps rather than on the justification of the overall mappings:

In Michalski's Inferential Learning Theory, non-deductive inference generates hypothesis to be confirmed.
Abduction and Induction is a collection of related works.
Inductive logic programming: from specific facts to general rules. Package in SWI-Prolog.

Learning a concept (specified by sufficient and necessary conditions) by generaling from (positive or negative) sample cases:

Version space search where hypotheses are revised by examples.
Decision tree learning acquires multiple concepts together by splitting the data by questions.

Representative projects on analogical reasoning:

Structure mapping engine (SME) maps between existing conceptual structures.
Copycat builds structures while establishing mapping between them.

3. Bayesian learning

Bayes' theorem is widely used as a learning rule, in which observed events are merged into background knowledge while a prior distribution is replaced by a posterior distribution.

Bayesian learning has been used to uniformly represent various types of inference, so as to gradually learn new knowledge of various types on complicated problems. There have been arguments on its potential in building artificial general intelligence, as well as in modeling the human brain and mind according to Free Energy Principle.

On the other hand, some limitations of Bayesian learning have been pointed out. Also, the request on consistency does not allow local updating, which is needed for real-time learning.

4. Learning and reasoning

With respect to the relation between learning and reasoning/inference, in AI there are three strategies:

Integrating a learning module and a reasoning module. E.g., neuro-symbolic models.
Using reasoning to carry out learning. E.g., "symbolic" learning approaches, Non-Axiomatic Logic.
Using learning to carry out reasoning. E.g., LLM, with various claims (1, 2) and criticisms (1, 2). A analysis of two types of "inference rules".

Readings

Luger: Sections 10.2-5, 7.3.2
Poole and Mackworth: Sections 10.2
Russell and Norvig: Sections 19.1, 19.3, 21.1-2