For the communication between AIs and human beings, Natural language processing (NLP) consists of understanding (NLU, comprehension) and generating (NLG, production).

NLP is more difficult than the processing of formal or programming languages, mainly because human languages were not designed with fixed syntax, semantics, and pragmatics, but instead evolved historically. As a result, these aspects (and the mappings among them) are never completely well-specified, change over time, and depend heavily on context and users

1. Symbolic (rule-based) approaches in NLP

1. Parsing: analyzing the syntactic structure of each sentence into a tree, according to a given grammar.
2. Interpreting: mapping a parsing tree to an internal representation, according to given definitions of the words.
3. Enhancing: augmenting the representation with background knowledge to get the intended meaning.
4. Using: supporting multiple usages of the represented sentences, according to given algorithms.

1. Content determination
2. Document structuring
3. Aggregation
4. Lexical choice
5. Referring expression generation
6. Realization

2. Statistical and neural approaches in NLP

1. Starting with a corpus, i.e., collections of actual text, with or without labels/annotations.
2. Learning relevant probability distributions,
3. Processing language (NLU and NLG) as probabilistic pattern association and prediction.

3. Word Embeddings

The input layer uses a "one-hot" encoding for words. There is an input unit for each word in a dictionary. For a given word, the corresponding unit has value 1, and the rest of the units have value 0. This input layer can feed into a hidden layer called an encoder, as it encodes each word into a vector. The embedding is mapped by a decoder to the output layer, where each node represents a word.

This neural network can be trained by backpropagation to learn the "context" of a word, i.e., the words appearing around it, such as predicting the next word (Example 8.9).

This model also captures some relations between words using vector operations (addition and subtraction), which can be used to answer analogical questions in the form of A:B = C:X, to pick a value for X, given A, B, and C.

4. Recurrent Neural Networks

Figure 8.13 shows an encoder–decoder recurrent neural network which does sequence-to-sequence mapping where the inputs and the outputs are not matched in a one-to-one manner. In this model, the hidden layer at the end of the encoder, c, becomes the context for the decoder and contains all of the information about the input sequence. The decoder is a generative language model that takes the context and emits an output sequence. The output sequence is produced one element at a time.

For a given input size, recurrent neural networks are feedforward networks with shared parameters. Stochastic gradient descent and related methods work, with the proviso that during backpropagation, the (shared) weights are updated whenever they were used in the prediction.

A Long Short-Term Memory network is a special kind of recurrent neural network designed to better capture long-term dependencies by maintaining the memory of previous states unless replaced by new information.

5. Transformers

Transformer models use an attention mechanism to decide the influence of each input node on an output node in the context to estimate the probability for a token to occur, selectively conditioned by the preceding tokens.

Large-Language Models (LLMs) are neural networks that are pretrained with a huge amount of language materials, and can be used for various NLP tasks. The basic function of an LLM is to predict the next token in the input stream, though in the process it has also acquired syntactic, semantic, and ontological knowledge embedded in the training materials, as well as showing various "Emergent Abilities".