**Growth of Functions**

A central topic in algorithm analysis is to compare the time (or space) efficiency of algorithms (for the same problem). As the running time of an algorithm is represented as a function of instance size, the problem is transformed into categorization of these functions.

- A
*size*measurement can be established among the (infinite number of) instances of a problem to indicate their*difficulty*, so that the*time*spent by an algorithm on an instance is a*nondecreasing function*of its size; - To compare the efficiency of algorithms on all problem instances, what matters is the instances larger than a certain size, as there are only finite smaller ones, but infinite larger ones;
- No matter where the threshold is set, in general the algorithm that grows faster will cost more than another one that grows slower;
- Since it is impossible to compare all growth functions, it is often enough to categorize their
*order of growth*using representative functions.

- Θ(g(n)) = {f(n) : There exist positive constants c
_{1}, c_{2}, and n_{0}such that 0 ≤ c_{1}g(n) ≤ f(n) ≤ c_{2}g(n) for all n ≥ n_{0}}. We say that g(n) is an*asymptotically tight bound*of f(n), and write f(n) = Θ(g(n)). [It actually means f(n) ∈ Θ(g(n)). The same for the following.] - O(g(n)) = {f(n) : There exist positive constants c and n
_{0}such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n_{0}}. We say that g(n) is an*asymptotically upper bound*of f(n), and write f(n) = O(g(n)). - Ω(g(n)) = {f(n) : There exist positive constants c and n
_{0}such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n_{0}}. We say that g(n) is an*asymptotically lower bound*of f(n), and write f(n) = Ω(g(n)). - o(g(n)) = {f(n) : For any positive constant c > 0, there exist a constant n
_{0}> 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n_{0}}. We say that g(n) is an*upper bound*of f(n) but*not asymptotically tight*, and write f(n) = o(g(n)). - ω(g(n)) = {f(n) : For any positive constant c > 0, there exist a constant n
_{0}> 0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n_{0}}. We say that g(n) is a*lower bound*of f(n) but*not asymptotically tight*, and write f(n) = ω(g(n)).

Examples:

- 4n
^{3}− 10.5n^{2}+ 7n + 103 = Θ(n^{3}) - 4n
^{3}− 10.5n^{2}+ 7n + 103 = O(n^{3}) - 4n
^{3}− 10.5n^{2}+ 7n + 103 = Ω(n^{3}) - 4n
^{3}− 10.5n^{2}+ 7n + 103 = O(n^{4}) - 4n
^{3}− 10.5n^{2}+ 7n + 103 = Ω(n^{2}) - 4n
^{3}− 10.5n^{2}+ 7n + 103 = o(n^{4}) - 4n
^{3}− 10.5n^{2}+ 7n + 103 = ω(n^{2})

Relations among the five:

- f(n) = Θ(g(n)) if and only if f(n) = O(g(n)) and f(n) = Ω(g(n)).
- f(n) = O(g(n)) if and only if f(n) = Θ(g(n)) or f(n) = o(g(n)).
- f(n) = Ω(g(n)) if and only if f(n) = Θ(g(n)) or f(n) = ω(g(n)).
- f(n) = O(g(n)) if and only if g(n) = Ω(f(n)).
- f(n) = o(g(n)) if and only if g(n) = ω(f(n)).

- f(n) = Θ(g(n)) if and only if Lim[f(n)/g(n)] = c (a positive constant).
- f(n) = o(g(n)) if and only if Lim[f(n)/g(n)] = 0.
- f(n) = ω(g(n)) if and only if Lim[f(n)/g(n)] = ∞.

- constant: Θ(
*1*) - logarithmic: Θ(log
),_{a}n*a*> 1 - polynomial: Θ(
*n*),^{a}*a*> 0 - exponential: Θ(
*a*),^{n}*a*> 1 - factorial: Θ(
*n!*)

Asymptotic notions can be used in equations and inequalities, as well as in certain calculations. For example,

2n^{2} + 3n + 1 = Θ(n^{2}) + Θ(n) + Θ(1) = Θ(n^{2}).

In algorithm analysis, the most common conclusions are worst case expenses expressed as O(f(n)), which can be obtained by focusing on the most expensive item in the function, as in the analysis of Insertion Sort algorithm.

In general, the "divide-and-conquer" approach uses *D(n)* time to divide a problem into *a* smaller subproblems of the same type, each with *1/b* of the original size. After the subproblems are solved, their solutions are combined in *C(n)* time to get the solution for the original problem. This process is repeated on the subproblems, until the input size becomes so small that the problem can be solved in constant time. This approach gives us the following recurrence:

For example, for the Merge Sort algorithm, we have

This is the case, because each time an array of size *n* is divided into two halves, and to merge the results together takes linear time.

Often we need to solve the recurrence, so as to get a running time function without recursive call.

One way to solve a recurrence is to use the *substitution method*, which uses mathematical induction to prove a function that was guessed previously.

For example, if the function is

one reasonable guess is T(n) = O(*n* lg *n*). To show that this is indeed the case, we can prove that T(n) ≤ *c n* lg *n* for an appropriate choice of the constant *c* > 0. We start by assuming that this bound holds for halves of the array, then, substituting into the recurrence yields,

where the last step holds as long as c ≥ 1.

Furthermore, mathematical induction requires us to show that our solutions holds for the boundary conditions, that is, we can choose the constant *c* large enough so that T(n) ≤ *c n* lg *n* holds there. For this example, if the boundary condition is T(1) = 1, then c = 1 does not work there because *c n* lg(n) = 1 1 lg(1) = 0. To resolve this issue we only need to show that for n = 2, we do have T(n) ≤ *c* 2 lg 2 as long as c ≥ 2.

In summary, we can use *c ≥ 2*, and we have shown that T(n) ≤ *c n* lg *n* for all *n* ≥ 2.

To get a good guess for recurrence function, one way is to draw a *recursion tree*. For example, if the function is

we can create the following recursion tree:

To determine the height of the tree *i*, we have *n* / 4^{i} = 1, that is, *i* = log_{4}*n*. So the tree has log_{4}*n* + 1 levels.

At level *i*, there are 3^{i} nodes, each with a cost *c*(n / 4^{i})^{2}, so the total is (3/16)^{i}cn^{2}.

At the leave level, the number of nodes is 3^{log4n}, which is equal to *n*^{log43} (see page 54). At that level, each node costs T(1), so the total is *n*^{log43}T(1), which is Θ(*n*^{log43}).

After some calculation (see Section 4.4 of the textbook for details), the guess T(n) = O(n^{2}) can be obtained.

Finally, there is a "master method" that provides a general solution for all divide-and-conquer recurrence. Intuitively, the theorem says that the solution to the recurrence is determined by the larger one of the costs at the top and bottom levels of the recursion tree, respectively:

The above example falls into Case 3.

The proof of the master theorem is given in the textbook. It is important to realize that the three cases of the master theorem do not cover all the possibilities.

As a special case, when *a = b* we have log* _{b}a* = 1. The above results are simplified to (1) Θ(