\documentclass[11pt]{article}

%\setlength{\topmargin}{-.8in}  
% \setlength{\oddsidemargin}{0in}
% \setlength{\evensidemargin}{0in}
% \setlength{\textwidth}{6.5in}
% \setlength{\textheight}{9.5in}

\usepackage{graphicx}
\usepackage{fullpage}
\begin{document}

\title{{\bf Mergesort and Recurrences}\\
\normalsize{(CLRS 2.3, 4.4)}}

\date{}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We saw a couple of $O(n^2)$ algorithms for sorting. Today we'll see a
different approach that runs in $O(n \lg n)$ and uses one of the most
powerful techniques for algorithm design, divide-and-conquer.

Outline: 
\begin{enumerate}
\item Introduce the divide-and-conquer algorithm technique.
\item Discuss a sorting algorithm obtained using divide-and-conquer (mergesort).
\item Introduce reccurences as a means to express the running time of
  recursive algorithms. 
\item Discuss iteration (recursion tree) as a way to solve a
  reccurrence. 

\end{enumerate}


\section{Divide-and-conquer }

Let's say we want to solve a problem $P$. For e.g. $P$ could be the
problem of sorting an array, or finding the smallest element in an
array. Divide-and-conquer is an approach that can be applied to any
$P$ and goes like this:

\vspace{1cm}
\fbox{
  \parbox{13cm}{{\bf Divide-and-Conquer}
    \\
    \\ To Solve P:
    \begin{enumerate}
    \item {\em Divide} P into two smaller problems $P_1,P_2$.
    \item {\em Conquer} by solving the (smaller) subproblems recursively.
    \item {\em Combine} solutions to $P_1,P_2$ into solution for P.
    \end{enumerate}
  }
}\\

\vspace{1cm}
The simplest way is to divide into \emph{two} subproblems.  Can be
extended to divide into $k$ subproblems.

Analysis of divide-and-conquer algorithms and in general of recursive
algorithms leads to recurrences.
  


\newpage
\section{MergeSort}

A divide-and-conquer solution for sorting an array gives an algorithm
known as mergesort:

\begin{itemize}
\item Mergesort: 
  \begin{itemize}
  \item Divide: Divide an array of $n$ elements into two arrays of $n/2$
    elements each.
  \item Conquer: Sort the two arrays recursively.
  \item Combine: Merge the two sorted arrays.
  \end{itemize}
  
  
\item Assume we have procedure Merge$(A,p,q,r)$ which merges sorted 
  A[p..q] with sorted A[q+1....r]
\item We can sort A[p...r] as follows (initially p=0 and r=n-1):
  
  \fbox{
    \parbox{5.5cm} { Merge Sort(A,p,r)
      \begin{itemize}
      \item[] If $p<r$ then
        \begin{itemize}
        \item[] $q =\lfloor(p+r)/2\rfloor$
        \item[] MergeSort(A,p,q)
        \item[] MergeSort(A,q+1,r)
        \item[] Merge(A,p,q,r)
        \end{itemize}
      \end{itemize}
    }}
 
  
\item How does Merge$(A,p,q,r)$ work? 
  \begin{itemize}
    
  \item Imagine merging two sorted piles of cards. The basic
    idea is to choose the smallest of the two top cards and put it into
    the output pile.
    
  \item Running time: $\Theta(r-p)$
    
  \item Implementation is a bit messier.
    
  \end{itemize}
  
\end{itemize}



\subsection{Mergesort Correctness}
\begin{itemize}
\item Merge: Why is merge correct? As you look at the next item to put
  into the merged output, what has to be true?
\item Assuming that Merge is correct, prove that Mergesort() is correct. 
  \begin{itemize}
  \item Visualize the recursion tree of mergesort, sorting comes to
    down to a bunch of merges. If merge works correctly, then
    mergesort works correctly.
  \item Formally, we need induction on $n$
  \end{itemize}
\end{itemize}




\subsection{Mergesort Example}
\vspace{2cm}
\includegraphics[width=8cm]{mergesort1.pdf} 





\newpage
\subsection{Mergesort Analysis}
\begin{itemize}
\item To simplify things, let us assume that $n$ is a power of 2, i.e
  $n = 2^k$ for some k.

\item Running time of a recursive algorithm can be analyzed using a
  {\bf recurrence relation}. Each ``divide'' step yields two
  sub-problems of size $n/2$.

\item Let $T(n)$ denote the worst-case running time of mergesort on an array of $n$ elements. We have: 

  \begin{eqnarray*}
  T(n) & =  & c_1 + T(n/2) + T(n/2) +c_2n \\
  & =  & 2T(n/2) + (c_1 + c_2n)
\end{eqnarray*}

\item Simplified, $T(n) = 2T(n/2) + \Theta(n)$

\item We can see that the recurrence has solution $\Theta(n\log_2 n)$ by
  looking at the {\bf recursion tree}: the total number of levels in
  the recursion tree is $\log_2 n + 1$ and each level costs linear
  time (more below).

\item Note: If $n \neq 2^k$ the recurrence gets more complicated.

  $T(n)=\left\{ \begin{array}{ll}
      \Theta(1) & \mbox{\rm If} ~n=1 \\
      T(\lceil \frac{n}{2}\rceil)+T(\lfloor
      \frac{n}{2}\rfloor)+\Theta(n) & \mbox{\rm
        If}  ~n>1
    \end{array} \right.$

  But we are interested in the order of growth, not in the exact
  answer.  So we first solve the simple version (equivalent to
  assuming that $n=2^k$ for some constant $k$, and leaving out base
  case and constant in $\Theta$).  Once we know the solution for the
  simple version, one needs to solve the original recursion by
  induction. This step is necessary for a complete proof, but it is
  rather mechanical, so it is usually skipped.

  So even if we are ``sloppy'' with ceilings and floors, the
  solution is the same. We usually assume $n=2^k$ or whatever to avoid
  complicated cases.

\end{itemize}




\section{Solving recurrences}

The steps for solving a recurrence relation are the following: 

\begin{enumerate}
\item Draw the recursion tree to get a feel for how the recursion
  goes. Sometimes, for easy recurrences, it is sufficient to see the
  bound. This step can be skipped.
\item Iterate and solve the summations to get the final bound.

\item Use induction to prove this bound formally (substitution
  method).
  
\end{enumerate}

In this incarnation of the class we will skip the induction step ----
generally speaking this step is pretty mechanical.

For us solving a recurrence will mean finding a theta-bound for $T(n)$
by iteration.



%\newpage
% \section{Solving recurrences by induction (the Substitution method)}

% \begin{itemize}
% \item Idea: Make a guess for the form of the solution and prove by
%   induction.
% \item Can be used to prove both upper bounds $O()$ and lower bounds $\Omega()$. 
% \item Let's solve $T(n)=2T(n/2)+n$ using substitution
%   \begin{itemize}
%   \item Guess $T(n)\leq cn\log n$ for some constant $c$ (that is,
%     $T(n)=O(n\log n$))
%   \item Proof:
%     \begin{itemize}
      
%     \item Base case: we need to show that our guess holds for some
%       base case (not necessarily $n=1$, some small $n$ is ok).  Ok,
%       since function constant for small constant $n$.

%     \item Assume holds for $n/2$: $T(n/2)\leq c\frac{n}{2}\log
%       \frac{n}{2}$ (Question: Why not $n-1$?)
      
%       Prove that holds for $n$: $T(n)\leq cn\log n$
      
%       \begin{eqnarray*}
%         T(n) & = & 2T(n/2)+n \\
%         & \leq & 2(c\frac{n}{2}\log \frac{n}{2})+n \\
%         & = & cn\log \frac{n}{2} +n \\
%         & = & cn\log n-cn\log 2 +n \\
%         & = & cn\log n-cn+n \\
%       \end{eqnarray*}

%      So ok if $c\geq 1$
%     \end{itemize}
%   \end{itemize}
  
% \item Similarly it can be shown that $T(n)=\Omega(n\log n)$
  
%   Exercise!
  
% \item Similarly it can be shown that $T(n) = T(\lfloor
%   \frac{n}{2} \rfloor) + T(\lceil \frac{n}{2} \rceil ) + n$ is
%   $\Theta(n \lg n)$.
  
%   Exercise!
  
% \item The hard part of the substitution method is often to make a good
%   guess. How do we make a good (i.e. tight) guess???  Unfortunately,
%   there's no ``recipe'' for this one.  Try iteratively $O(n^3),
%   \Omega(n^3), O(n^2), \Omega(n^2)$ and so on. Try solving by
%   iteration to get a feeling of the growth.

% \end{itemize}




\newpage
\subsection{Solving Recurrences via Recursion tree}


\begin{itemize}
  
\item We draw out the recursion tree with cost of single call in each
  node---running time is sum of costs in all nodes 
  
\item If you are careful drawing the recursion tree and summing up
  the costs, the recursion tree  gives you the solution  of
  the recurrence
  
\item Example: $T(n)=8T(n/2)+n^2$~~($T(1)=1$)
  
  \vfill
  
  \hspace*{-.25in}
  \includegraphics[width=6in]{iteration.pdf}
  
  \vfill
  
  $T(n) = n^2 + 2n^2 + 2^2n^2 + 2^3n^2 + 2^4n^2 + \dots + 2^{\log n -1}n^2
  + 8^{\log n} = .... = \Theta(n^3)$
\end{itemize}




\newpage
\section{Solving Recurrences by iteration}

\begin{itemize}

\item Example: Solve $T(n)=8T(n/2)+n^2$~~(with $T(1)=1$)

\begin{eqnarray*}
  T(n) & = & n^2 + 8T(n/2) \\
  & = & n^2 + 8(8T(\frac{n}{2^2})+(\frac{n}{2})^2) \\
  & = & n^2 + 8^2T(\frac{n}{2^2})+8(\frac{n^2}{4})) \\
  & = & n^2 + 2n^2 + 8^2T(\frac{n}{2^2}) \\
  & = & n^2 + 2n^2 + 8^2(8T(\frac{n}{2^3})+(\frac{n}{2^2})^2) \\
  & = & n^2 + 2n^2 + 8^3T(\frac{n}{2^3})+8^2(\frac{n^2}{4^2})) \\
  & = & n^2 + 2n^2 + 2^2n^2 + 8^3T(\frac{n}{2^3}) \\
  & = & \dots \\
  & = & n^2 + 2n^2 + 2^2n^2 + 2^3n^2 + 2^4n^2 + \dots 
\end{eqnarray*}

\begin{itemize} 
  
\item Recursion depth: How long (how many iterations) it takes until
  the subproblem has constant size?  $i$ times where
  $\frac{n}{2^i}=1 \Rightarrow i=\log n$
  
\item What is the last term? $8^iT(1)=8^{\log n}$ \end{itemize}

\begin{eqnarray*}
  T(n) & = & n^2 + 2n^2 + 2^2n^2 + 2^3n^2 + 2^4n^2 + \dots + 2^{\log n -1}n^2 +
  8^{\log n} \\
  & = & \sum_{k=0}^{\log n-1} 2^kn^2 + 8^{\log n} \\
  & = & n^2 \sum_{k=0}^{\log n-1} 2^k + (2^3)^{\log n}
\end{eqnarray*}

\item Now $\sum_{k=0}^{\log n-1} 2^k$ is a geometric sum so we have
  $\sum_{k=0}^{\log n-1} 2^k=\Theta(2^{\log n-1})=\Theta(n)$
  
\item $(2^3)^{\log n}=(2^{\log n})^3=n^3$
  
  \begin{eqnarray*}
T(n) & = & n^2 \cdot \Theta(n) + n^3 \\
     & = & \Theta(n^3)
\end{eqnarray*}
\end{itemize}






\newpage
\section{Other recurrences}

Some important/typical bounds on recurrences not covered by master method:

\begin{itemize}
\item Logarithmic: $\Theta(\log n)$
  \begin{itemize}
  \item Recurrence: $T(n)=1+T(n/2)$
  \item Typical example: Recurse on half the input (and throw half away)
  \item Variations: $T(n)=1+T(99n/100)$  
\end{itemize}
\item Linear: $\Theta(N)$
  \begin{itemize}
  \item Recurrence: $T(n)=1+T(n-1)$
  \item Typical example: Single loop
  \item Variations: $T(n)=1+2T(n/2), T(n)=n+T(n/2), T(n)=T(n/5)+T(7n/10+6)+n$
  \end{itemize}
\item Quadratic: $\Theta(n^2)$
  \begin{itemize}
  \item Recurrence: $T(n)=n+T(n-1)$

  \item Typical example: Nested loops
  \end{itemize}
\item Exponential: $\Theta(2^n)$
  \begin{itemize}
  \item Recurrence: $T(n)=2T(n-1)$
  \end{itemize}
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%








%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Optional} 

\subsection{Master Method}

\begin{itemize}
\item It is possible to come up with a formula for recurrences of the
  form $T(n) = aT(n/b) + n^c~~$ ($T(1)=1$). This is called the \emph{master method}. 
  \begin{itemize}
  \item Merge-sort $\Rightarrow T(n)=2T(n/2)+n$ ($a=2, b=2$, and $c=1$).
  \end{itemize}
  
  \fbox{
    \parbox{7cm}{
      $T(n) = aT\left(\frac{n}{b}\right) + n^c~~~a\geq1, b\geq1, c>0 \\
      \Downarrow \\
      T(n) = \left\{ \begin{array}{ll}
          \Theta(n^{\log_b a}) & a>b^c \\ 
          \Theta(n^c \log_b n) & a=b^c \\
          \Theta(n^c)          & a<b^c
        \end{array}
      \right.$
    }}
\end{itemize}


\vspace{\baselineskip}

Proof (by iteration )\\

$\begin{array}{lcl}
T(n) &=&  aT\left(\frac{n}{b}\right) + n^c \\
     &=&  n^c + a\left( \left(\frac{n}{b}\right)^c + aT\left(\frac{n}{b^2}\right) \right) \\
     &=&  n^c + \left(\frac{a}{b^c}\right)n^c + a^2T\left(\frac{n}{b^2}\right) \\
     &=&   n^c + \left(\frac{a}{b^c}\right)n^c + a^2\left( \left(\frac{n}{b^2}\right)^c + aT\left(\frac{n}{b^3}\right) \right) \\
     &=&  n^c + \left(\frac{a}{b^c}\right)n^c + \left(\frac{a}{b^c}\right)^2 n^c + a^3T\left(\frac{n}{b^3}\right) \\
     &=&  ... \\
     &=&  n^c + \left(\frac{a}{b^c}\right)n^c + \left(\frac{a}{b^c}\right)^2 n^c + \left(\frac{a}{b^c}\right)^3 n^c + \left(\frac{a}{b^c}\right)^4 n^c + ... + \left(\frac{a}{b^c}\right)^{\log_b n -1} n^c + a^{\log_b n}T(1)  \\
     &=&  n^c \sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k +  a^{\log_b n} \\
     &=&  n^c \sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k +  n^{\log_b a}
\end{array}$

\vspace{\baselineskip}

Recall geometric sum $\sum_{k=0}^n x^k = \frac{x^{n+1}-1}{x-1} = \Theta(x^n)$


\begin{itemize}
\item \fbox{$a<b^c$}

$a<b^c \Leftrightarrow \frac{a}{b^c}<1 \Rightarrow 
\sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k \le
\sum_{k=0}^{+\infty} \left(\frac{a}{b^c}\right)^k =
\frac{1}{1-\left(\frac{a}{b^c}\right)} = \Theta(1) \\ 
a<b^c \Leftrightarrow \log_b a<\log_b b^c =c\\
\begin{array}{lcl}
T(n) &=& n^c \sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k +  n^{\log_b a} \\
     &=& n^c\cdot \Theta(1) + n^{\log_b a} \\
     &=& \Theta(n^c)
\end{array}$

\item \fbox{$a=b^c$}

$a=b^c \Leftrightarrow \frac{a}{b^c}=1 \Rightarrow  
\sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k = \sum_{k=0}^{\log_b
n-1} 1 = \Theta(\log_b n) \\
a=b^c \Leftrightarrow \log_b a=\log_b b^c =c\\
\begin{array}{lcl}
T(n) &=&  \sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k +  n^{\log_b a} \\ 
     &=&  n^c \Theta(\log_b n) +  n^{\log_b a} \\
     &=&  \Theta( n^c \log_b n ) \\
\end{array} $

\item \fbox{$a>b^c$}

$ a>b^c \Leftrightarrow \frac{a}{b^c}>1 \Rightarrow 
\sum_{k=0}^{\log_b n -1} \left(\frac{a}{b^c}\right)^k =  
\Theta\left(\left(\frac{a}{b^c}\right)^{\log_b n}\right) = 
\Theta\left(\frac{a^{\log_b n}}{(b^c)^{\log_b n}}\right) = 
\Theta\left(\frac{a^{\log_b n}}{n^c}\right)  \\
\begin{array}{lcl}
T(n) &=&  n^c\cdot \Theta\left(\frac{a^{\log_b n}}{n^c}\right) +  n^{\log_b a} \\ 
     &=&  \Theta( n^{\log_b a} ) +  n^{\log_b a} \\
     &=&  \Theta( n^{\log_b a} ) \\
\end{array} $

\item Note: Book states and proves the result slightly differently.
\end{itemize}





\subsection{Changing variables}

Sometimes reucurrences can be reduced to simpler ones by {\em changing variables\/}

   \begin{itemize}
   \item Example: Solve $T(n)=2T(\sqrt{n})+\log n$

   \vspace{\baselineskip}

   Let $m=\log n \Rightarrow 2^m=n \Rightarrow \sqrt{n}=2^{m/2}$

   $T(n)=2T(\sqrt{n})+\log n \Rightarrow T(2^m)=2T(2^{m/2})+m$

   \vspace{\baselineskip}

   Let $S(m)=T(2^m)$

   $T(2^m)=2T(2^{m/2})+m \begin{array}[t]{l}
      \Rightarrow S(m)=2S(m/2)+m \\
      \Rightarrow S(m)=O(m\log m) \\
      \Rightarrow T(n)=T(2^m)=S(m)=O(m\log m)=O(\log
      n\log \log n)
      \end{array}$
   \end{itemize}








\end{document}
