CIS 5511. Programming Techniques
Greedy Algorithms
1. The idea
A greedy algorithm is a problem-solving technique often used in optimization problems. At each step, the algorithm makes the locally optimal choice, without reconsidering previous decisions. The hope is that a sequence of locally optimal choices will lead to a globally optimal solution.
Greedy algorithms are usually simple and efficient because they avoid exploring all possible combinations of choices. However, a locally optimal decision does not always contribute to a globally optimal solution.
For example, in the planning matrix-chain multiplication problem discussed previously, a greedy strategy may repeatedly choose the multiplication that appears to reduce the largest matrix dimension. Although this choice seems beneficial at the current step, it can prevent the algorithm from finding the globally optimal parenthesization. In the second example, this algorithm produces the solution ((A1*A2)*A3)*(A4*(A5*A6)), which is not optimal.
Unlike dynamic programming, a greedy algorithm typically does not compare all relevant alternatives or remember previously computed results. Consequently, many greedy algorithms do not always find an optimal solution, though some of them do.
2. Example: Activity Selection
Problem: for a given set of activities A1 ... An, each with its start and finish time for using a resource exclusively, find a schedule that maximizes the number of compatible activities. Please note that this is not the same as maximizing the time for the resource to be used.
A greedy solution: find the activity with the earliest finish time that is consistent with the current schedule, add it into the schedule, then recursively create the rest of the schedule.
In the following algorithm, array s and f represent the start and finish time of the activities, and they are sorted in monotonically increasing order of finish time:
Beside the two arrays, the algorithm also takes indexes k and n as input, to indicate the range of activities to be considered. Initially, it is called as Recursively-Activity-Selector(s, f, 0, n) by algorithm Activity-Selector(s, f).
The recursive algorithm can be converted into a more efficient iterative algorithm:
Both algorithms take Θ(n) time.
The optimality of this algorithm follows from an exchange argument. Any optimal schedule can be transformed into another optimal schedule that contains the activity with the earliest finish time. After selecting this activity, the remaining problem has the same form as the original one.
3. Example: Knapsack problem
Given a set of items, each with a weight wi and a value vi, determine the number of each item to include in a collection so that the total weight is less than or equal to a given limit W and the total value is as large as possible.
If it is allowed to take a fraction of an item, the problem is called "fractional knapsack problem" and a greedy algorithm can find an optimal solution by repeatedly picking the item with the highest unit value until that item is exhausted or W has been reached.
If every item must be completely taken or left, the problem is called "0-1 knapsack problem" and no greedy algorithm (such as the previous one, or repeatedly picking the item with the highest value) can always find an optimal solution. Instead, dynamic programming should be used for that.
Example:
This example illustrates an important lesson: a greedy strategy may be optimal for one version of a problem (fractional knapsack) but not for a closely related version (0-1 knapsack).
4. Example: Huffman coding
Huffman coding is an efficient way to compress data for communication and storage. The idea is to use variable-length code for data units, where those with higher frequencies have shorter codes. There is no ambiguity as long as no codeword is also a prefix of some other codeword.
In the following example, the variable-length codeword has an average of 2.24 bits:
Assuming binary prefix coding, a solution can be represented by a binary tree, with each leaf corresponds to an unit, and its path from root represents the code for the unit (left as 0, right as 1). An optimal code is a binary tree with the minimum expected path length from the root to the leaves.
In the following algorithm, C contains a set of units, and Q is a priority queue containing binary trees prioritized by the frequency of their roots.
Procedure HUFFMAN produces an optimal prefix code. Example:
Huffman coding is one of the most important examples of a greedy algorithm. The repeated selection of the two least frequent symbols can be shown to be part of some optimal prefix code, which guarantees the correctness of the algorithm.
5. Related techniques
Additional common greedy algorithms:
- Hill Climbing: When an optimization problem can be represented as searching for the maximum value of a function in a multidimensional space, "hill climbing" is a greedy algorithm which attempts to find a better solution by making an incremental change to the current solution.
- Gradient Descent: The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent.
- Best-First Search: If the next step is selected at the neighborhood of all explored points, "hill climbing" becomes "best-first search". As the selection is based on a heuristic function, it is also called heuristic search.
6. Alternative quality-cost tradeoffs
When an in-time solution is preferred over an optimal solution, several approaches besides greedy algorithms can be employed to balance solution quality against computational cost:
- Real-time Computing: The response time is included in the specification of each problem instance. Time restrictions can be "hard" (as deadline) or "soft" (as degrading utility). The solution often depends hardware factors.
- Approximation algorithms: Expanding the scope of acceptable solutions beyond the optimal one.
- Flexible Computation: Decide which algorithm (or how much computation) to use given the available time and resources.
- Anytime Algorithm: Such an algorithm improves the quality of its solution over time, and can stop at anytime with a best-so-far solution (e.g., approximating π). An anytime algorithm no longer has a determined solution and running time for a given problem instance.
- Case-by-Case Problem Solving: To solve each occurrence, or case, of a problem using currently available knowledge and resources on the case. It is different from the traditional Algorithmic Problem Solving, which applies the same algorithm to all occurrences of all problem instances. Case-by-case Problem Solving is suitable for situations where the system has no applicable algorithm for a problem as a whole, while still may solve some special cases of it. This approach gives the system flexibility, originality, and scalability, at the cost of predictability, repeatability, and terminability.