5511-10

CIS 5511. Programming Techniques

Dynamic Programming

1. The idea

When a problem is solved by reducing it to subproblems, sometimes the same subproblem appears multiple times.

For example, Fibonacci numbers are defined by the following recurrence:

    F(0) = 0
    F(1) = 1
    F(i) = F(i-1) + F(i-2)

A recursive algorithm derived from this definition is:

    Fibonacci-R(i)
    1  if i == 0
    2      return 0
    3  if i == 1
    4      return 1
    5  return Fibonacci-R(i-1) + Fibonacci-R(i-2)

This algorithm contains repeated calculations and is therefore inefficient. For example, to get F(4), it calculates F(3) and F(2) separately, though in calculating F(3), F(2) is already calculated. The time complexity of this algorithm is O(2ⁱ).

This algorithm can be rewritten in a "bottom-up" manner with the intermediate results saved in an array. This gives us a "dynamic programming" version of the algorithm:

Fibonacci-DP(i)
    1  A[0] = 0        // assume 0 is a valid index
    2  A[1] = 1
    3  for j = 2 to i
    4      A[j] = A[j-1] + A[j-2]
    5  return A[i]

Since each subproblem solution is computed only once, the time complexity is O(i).

This example illustrates the basic idea of dynamic programming: although the solution is defined in a top-down manner (from a problem to its subproblems), it is constructed in a bottom-up manner (from solutions of subproblems to the full solution), while intermediate results are stored to avoid repeated calculations.

Memoization is a variation of dynamic programming that keeps the efficiency, while maintaining a top-down structure.

    Fibonacci-M(i)
    1  A[0] = 0        // assume 0 is a valid index
    2  A[1] = 1
    3  for j = 2 to i
    4      A[j] = -1
    5  Fibonacci-M2(A, i)

    Fibonacci-M2(A, i)
    1  if A[i] < 0
    2      A[i] = Fibonacci-M2(A, i-1) + Fibonacci-M2(A, i-2)
    3  return A[i]

The time complexity of this algorithm is also O(i).

Dynamic programming is often used in optimization problems in which an optimal solution corresponds to a particular decomposition of the problem into subproblems. When different ways of division are compared, it is quite common that some subproblems are involved multiple times, therefore dynamic programming provides a more efficient algorithm than (direct) recursion.

The difficulty of implementing dynamic programming arises from the need to keep intermediate results and to record the structure of the optimal solution, i.e., where each partial solution comes from.

2. Planning matrix-chain multiplication

The following procedure multiplies two matrices A (p × q) and B (q × r) to get a matrix C (p × r) such that C = AB.

The result C contains p×r elements, each calculated from q scalar multiplications and q − 1 additions. If we count the number of scalar multiplications and use it to represent running time, then for such an operation, it is p×q×r.

To compute the product of a matrix chain A₁A₂...A_n, exactly n − 1 matrix multiplications are required. Although matrix multiplication is associative (i.e., allowing any adjacent pair to be multiplied at each step), different evaluation orders yield significantly different computational costs

Taking matrix-chain A₁A₂A₃ as an example, if the three have sizes 10×2, 2×20, and 20×5, respectively, then the cost of (A₁A₂)A₃ is 10×2×20 + 10×20×5 = 1400, while the cost of A₁(A₂A₃) is 2×20×5 + 10×2×5 = 300.

In general, for matrix-chain A_i...A_j where each A_k has dimensions P_k-1×P_k, the minimum cost of the product m[i,j] can be obtained by looking for the best way to cut it into A_i...A_k and A_k+1...A_j. Therefore, we have

    m[i, j] = 0                                                if i = j
              min{ m[i,k] + m[k+1,j] + P_i-1P_kP_j }               if i < j

Since certain values of m[i,j] are reused many times, dynamic programming can be used to find the minimum number of operations by calculating this m value for sub-chains with increasing length, with another matrix s to keep the cutting point k for each m[i,j].

Please note that this algorithm determines the optimal order of matrix multiplications without actually performing the multiplications themselves.

Example: Given the following matrix dimensions:
A₁ is 30 × 35
A₂ is 35 × 15
A₃ is 15 × 5
A₄ is 5 × 10
A₅ is 10 × 20
A₆ is 20 × 25
P: [30, 35, 15, 5, 10, 20, 25]

At the end of the execution, the elements above the main diagonal in each matrix are (rotated 45 degrees counterclockwise in the figure):

which means that A₁A₂A₃A₄A₅A₆ should be calculated as (A₁(A₂A₃))((A₄A₅)A₆), and the number of scalar multiplications is 15,125.

This problem can be solved by either a bottom-up dynamic programming algorithm or a top-down, memoized dynamic-programming algorithm. The running time of both algorithms is Θ(n³), and the algorithm needs Θ(n²) space.

3. Longest Common Subsequence

A subsequence of a sequence can be obtained by removing zero or more elements while preserving the order of the remaining elements.

For sequences X and Y, their Longest Common Subsequence (LCS) is not necessarily contiguous in either X or Y. For example, if X is ABCBDAB and Y is BDCABA, a LCS of them is BCBA with length 4, and BCAB is another one. So the LCS is not unique, though their lengths are the same.

LCS can be recursively constructed. Let c[i, j] denote the length of a LCS of the sequences X₁ ... X_i and Y₁ ... Y_j, then

Algorithm (where the three arrowheads can be replaced by any distinct constants):

Time cost: Θ(m×n). Space cost: Θ(m×n).

Example:

4. Additional examples

Optimal Binary Search Tree: While a balanced binary search tree (BST) guarantees low worst-case search times, it does not necessarily minimize the average search cost when keys are accessed with non-uniform probabilities. In such cases, dynamic programming can be used to construct an optimal BST by systematically evaluating the expected search cost associated with every possible root, an approach that is structurally analogous to matrix-chain multiplication. Given the probabilities of successful searches for each key and unsuccessful searches for the intervals between keys, the algorithm determines the tree structure that minimizes the overall expected search cost.
Value Iteration and Policy Iteration are dynamic programming methods for solving Markov Decision Processes (MDPs) by iteratively refining value functions or policies until they converge to an optimal policy. Unlike traditional bottom-up dynamic programming algorithms that require only a single pass through the state space, MDP solution methods may revisit and update the same states multiple times because state transitions can form cycles, including self-transitions and returns to previously visited states.