\documentclass[11pt]{article}
\usepackage{fullpage}
\usepackage{graphics,epsfig}



\begin{document}

\title{{\bf Quicksort}\\
\normalsize{(CLRS 7)}}

\date{}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\begin{itemize}
\item We saw the divide-and-conquer technique at work resulting in
  Mergesort.  Mergesort summary:
   \begin{itemize}
   \item Partition $n$ elements array $A$ into two subarrays of $n/2$ elements
   each
   \item Sort the two subarrays recursively
   \item Merge the two subarrays
   \end{itemize}

   Running time: $T(n)=2T(n/2)+\Theta(n) \Rightarrow T(n)=\Theta(n\log
   n)$. Mergesort is not in-place.

\item Another possibility is to divide the elements such that there is
no need of merging, that is
   \begin{itemize}
   \item Partition $A[1...n]$ into subarrays $A'=A[1..q]$ and
   $A"=A[q+1...n]$ {\bf such that all elements in $A"$ are larger than all
   elements in $A'$}.
   \item Recursively sort $A'$ and $A"$.
   \item (no need to to combine/merge. $A$ already sorted after sorting
     $A'$ and $A"$)
   \end{itemize}

 \item Thus the pseudo code for {\sc Quicksort} would look like
   this. As usual with recursive functions on arrays, we see the array
   indices $p$ and $r$ as arguments.

\fbox{
\parbox{5cm}{
{\sc Quicksort}($A,p,r$) \\
IF $p<r$ THEN
\begin{itemize}
\item[] q={\sc Partition}($A,p,r$)
\item[] {\sc Quicksort}($A,p,q-1$)
\item[] {\sc Quicksort}($A,q+1,r$)
\end{itemize}
%FI
}}\\

To sort the entire array, one would need to call  {\sc Quicksort}($A,0,n-1$).

\vspace{\baselineskip}

\item Let's ignore for the moment how Partition would actually work
  ----we'll do that in a second. What can we expect in terms of
  running time? If $q=n/2$ and we partition in $\Theta(n)$ time, we
  again get the recurrence $T(n)=2T(n/2)+\Theta(n)$ for the running
  time $\Rightarrow T(n)=\Theta(n\log n)$

  The problem is: can we develop a partition algorithm which divides
  $A$ in (approximately) two halves?

\vspace{\baselineskip}
\item Again, let's ignore for the moment how Partition would actually
  work ----we'll do that in a second. Why is {\sc Quicksort} correct?
  Can be shown inductively, assuming that {\sc Partition} works
  correctly (which can be shown separately). Basically, if Partition
  works as expected and Quicksort sorts correctly the two sides of the
  partition (this is the inductive hypothesis), then it is easy to see
  that the whole array is sorted correctly. For those of you who know
  induction, this should be pretty straightfoward. For those of you
  who don;t know induction, either do some vigorous hand waving, or
  better, read up on induction -- there are some great links on the
  class website.
\end{itemize} 



\section*{Partition} 

Now that we got the big picture, let's dive into how to partition.  

\begin{itemize} 
\item First, note that it would be pretty eay to partition if we did not
have to do it in place (think, think..). 

\item The trick is doing it in-place. If we are able to come up with
  an in-place partition, this immediately leads to in-place quicksort,
  which is nice.

\item Below is the code from the textbook (CLRS). It is called {\sc
    Lomuto} partition, from the one who first described it.


\fbox{
\parbox{6cm}{
{\sc Partition}($A,p,r$) \\
$x=A[r]$ \\
$i=p-1$ \\
FOR $j=p$ TO $r-1$ DO 
\begin{itemize}
\item[] IF $A[j]\leq x$ THEN
	\begin{itemize}
	\item[] $i=i+1$
	\item[] Exchange $A[i]$ and $A[j]$
	\end{itemize}
\item[] FI	
\end{itemize}
OD\\
Exchange $A[i+1]$ and $A[r]$\\
RETURN $i+1$
}}
  


\item Example:
  
  \includegraphics[width=5cm]{quicksorteks.pdf}
  
\item{\sc Partition} can be proved correct  using
  the loop invariants:
  \begin{itemize}
  \item $A[k]\leq x$ for $p\leq k\leq i$
  \item $A[k]> x$ for $i+1\leq k\leq j-1$
  \item $A[k]=x$ for $k=r$
  \end{itemize}
  These are true before the execution of the loop, when $i=p-1, j=p$;
  and are true after every execution of the loop.  Thus at the end
  when $j=p-1$ this means partition works correctly. Formally one
  woudl need to use induction.

\item {\sc Partition} analysis: {\sc Partition} runs in time
  $\Theta(r-p)$ (does one pass through the input), which is linear in
  the number of elemenst that are partitioned.




\end{itemize}



\vspace{\baselineskip}
\section*{{\sc quicksort}  analysis}

\begin{itemize}
  
\item Running time depends on how well {\sc Partition} divides $A$.
\item In the example above it does reasonably well.
  
\item If array is always partitioned nicely in two halves (partition
  returns $q=\frac{r-p}{2}$), we have the recurrence $T(n) =
  2T(n/2) + \Theta(n) \Rightarrow T(n) = \Theta(n\lg n)$.
		
		
\item But, in the worst case {\sc Partition} always returns $q = p$
  or $q=r$ and the running time becomes $T(n)=\Theta(n)+T(0)+T(n-1)
  \Rightarrow T(n)=\Theta(n^2)$.
\item And maybe even worse, the worst case is when $A$ is already
  sorted.
  
\item Let's dig a step further. So all-good-splits give us
  $\Theta(n\lg n)$ and all-bad-splits give us $\Theta(n^2)$. What
  about a combination of good and bad splits?

\item Note that a good split is not necessarily a perfectly balanced
  one ($\frac12$-to-$\frac12$). Any split that puts a constact fraction of the
  elements to one side is good.
    \begin{itemize} 
    \item Example: Split is $\frac{9}{10}n$, $\frac{1}{10}n$.
      
      $T(n) = T(\frac{9}{10} n) + T(\frac{1}{10} n) + n$
      
      Solution: $\Theta(n \lg n)$
      
    % Guess: $T(n) \leq cn\log n$
    
    % Induction
% 		  \begin{eqnarray*}
% 		    T(n) & = & T(\frac{9}{10} n) + T(\frac{1}{10} n) + n \\
% 		    & \leq & \frac{9cn}{10}\log (\frac{9n}{10}) +
% 		    \frac{cn}{10}\log (\frac{n}{10}) + n \\
% 		    & \leq & \frac{9cn}{10}\log n + \frac{9cn}{10} \log
% 		    (\frac{9}{10}) + \frac{cn}{10}\log n + \frac{cn}{10}\log
% 		    (\frac{1}{10}) + n \\
% 		    & \leq & cn\log n + \frac{9cn}{10} \log 9 - \frac{9cn}{10}
% 		    \log 10 - \frac{cn}{10}\log 10 + n \\
% 		    & \leq & cn\log n - n(c\log 10 - \frac{9c}{10} \log 9 - 1) \\
% 		  \end{eqnarray*}
		  
% 		  $ T(n) \leq  cn\log n$ if $c\log 10 - \frac{9c}{10} \log 9
% 		  - 1 > 0 $ which is definitely true if $c>\frac{10}{\log 10}$
% 		\end{itemize}
		
% 	      \item So, in other words, if the splits happen at a constant
% 		fraction of $n$ we get $\Theta(n\lg n)$---or, it's almost
% 		never bad!
		
% 	      \end{itemize}
    \end{itemize} 

  \item So splits like $\frac12$-to-$\frac12$, $\frac13$-to-$\frac23$, ,,,
    $\frac19$-to-$\frac89$,.... are all good. There are a LOT of good
    splits!!!
  \item Even if every only other split is good, we can show that
    Quicksort still performs well (a bad split is absorbed into a good
    split).

  \item Intuitively, there are A LOT of cases where quicksort will
    perform in $\Theta(n \lg n)$ time. Can we theoretically justify
    this?
\end{itemize}




\section*{Average running time}

Formally, we want to find what is the average case running time of
{\sc quicksort}.  Is it close to worst-case ($\Theta(n^2)$, or to the
best case $\Theta(n\lg n)$?  We have to be careful here with the set
of inputs on which we take the average, because average time depends
on the distribution of inputs for which we take the average.
\begin{itemize}
\item If we run {\sc quicksort} on a set of inputs that are all almost
  sorted, the average running time will be close to the worst-case.
  
\item Similarly, if we run {\sc quicksort} on a set of inputs that
  give good splits, the average running time will be close to the
  best-case.
  
\item Most of the time, people analyse average time assuming a uniform
  distribution of the input. 
\end{itemize}

What happens if we run {\sc quicksort} on a set of inputs which are picked
uniformly at random from the space of all possible input permutations? 
\begin{itemize} 
\item Then the average case will also be close to the best-case.  Why?
  Intuitively, if any input ordering is equally likely, then we expect
  at least as many good splits as bad splits, therefore on the average
  a bad split will be followed by a good split, and it gets
  ``absorbed'' in the good split.
\end{itemize} 

So, under the assumption that {\bf all input permutations are equally
  likely}, the average time of {\sc Quicksort} is $\Theta(n\lg
n)$. 

\begin{itemize} 
\item This can be proved formally, but we won't do it in this class.
\end{itemize} 


Is is realistic to assume that all input permutations are equally
likely?

\begin{itemize}
\item Not really. In many cases the input is almost sorted (e.g. rebuilding index in a database etc).
\end{itemize}


If the assumption of uniform distribution is not realistic, then the
fact that Quicksort has a good average time does not help us that
much.  How can we make {\sc Quicksort} have a good average time
irrespective of the input distribution?

YES, using {\bf randomization}!! See below. 






\section*{Randomization}


We consider what are called \emph{randomized algorithms}, that is,
algorithms that make some random choices during their execution.

  \begin{itemize}
  \item Running time of a normal \emph{deterministic} algorithm only
  depends on the input.
  \item Running time of a randomized algorithm depends not only on
  input but also on the random choices made by the algorithm.
  \item Running time of a randomized algorithm is not fixed for a given input!

\item Randomized algorithms have best-case and worst-case running
times, but the inputs for which these are achieved are not known, they
can be any of the inputs.
  \end{itemize}

  We are normally interested in analyzing the \emph{expected} running
  time $T_e(n)$ of a randomized algorithm, that is, the expected
  (average)
  running time for all inputs of size $n$. Here $T(X)$ denotes the running time on input $X$ (of size $n$) \\

  \centerline{$T_e(n)=E_{|X|=n}[T(X)]$}



\section*{Randomized Quicksort}

There are two ways to go about randomizing Quicksort. 

\begin{enumerate}
\item The first one is: We can enforce that all $n!$ permutations are
  equally likely by randomly permuting the input before the algorithm.

\begin{itemize}
\item Most computers have pseudo-random number generator
  $random(1,n)$ returning ``random'' number between $1$ and $n$


\item Using pseudo-random number generator we can generate a random
  permutation (such that all $n!$ permutations equally likely) in
  $O(n)$ time:
  
  Choose element in $A[1]$ randomly among elements in $A[1..n]$,
  choose element in $A[2]$ randomly among elements in $A[2..n]$, 
  choose element in $A[3]$ randomly among elements in $A[3..n]$,
  and so on.
  
%   Note: Just choosing $A[i]$ randomly among elements $A[1..n]$ for
%   all $i$ will not give random permutation! Why?
\end{itemize}



\item Alternatively,  we can modify {\sc Partition} slightly and exchange last
element in $A$ with random element in $A$ before partitioning.

\fbox{
\parbox{5cm}{
{\sc RandPartition}($A,p,r$) \\
$i$={\sc Random}($p,r$) \\
Exchange $A[r]$ and $A[i]$ \\
RETURN {\sc Partition}($A,p,r$)
}}
%%\vspace{\baselineskip}
\fbox{
\parbox{6.5cm}{
{\sc RandQuicksort}($A,p,r$) \\
IF $p<r$ THEN
\begin{itemize}
\item[] q={\sc RandPartition}($A,p,r$)
\item[] {\sc RandQuicksort}($A,p,q-1$)
\item[] {\sc RandQuicksort}($A,q+1,r$)
\end{itemize}
}}\\

\end{enumerate}



It can be shown formally that the expected running time of randomized
quicksort (on inputs of size $n$) is $\Theta(n \lg n)$.
\begin{itemize} 
\item We do not do it in this class, but I encourage you to read the
  text book!
\end{itemize} 


% \section*{Expected Running Time of Randomized Quicksort}

% Let $T(n)$ be the running time of {\sc RandQuicksort} for an input of
% size $n$.


% \begin{itemize}
% \item Running time of {\sc RandQuicksort} is the total running time
%   spent in all {\sc Partition} calls.
% \item {\sc Partition} is called $n$ times
%   \begin{itemize}
%   \item The pivot element $x$ is not included in any recursive calls. 
%   \end{itemize}
% \item One call of {\sc Partition} takes $O(1)$ time plus time proportional
%  to the number of iterations of FOR-loop.
%   \begin{itemize}
%   \item In each iteration of FOR-loop we compare an element with the pivot
%   element.
%   \end{itemize}

% $\Downarrow$

% If $X$ is the number of comparisons $A[j]\leq x$ performed in {\sc
% Partition} over the entire execution of {\sc RandQuicksort} then the
% running time is $O(n+X)$.

% $\Downarrow$

% $E[T(n)] = E[O(n+X)] = n + E[X]$

% $\Downarrow$

% To analyze the expected running time we need to compute $E[X]$

% \item To compute $X$ we use $z_1, z_2, \dots, z_n$ to denote the elements
%   in $A$ where $z_i$ is the $i$th smallest element. We also use $Z_{ij}$
%   to denote $\{z_i,z_{i+1},\dots,z_j\}$.
% \item Each pair of elements $z_i$ and $z_j$ are compared at most once
%   (when either of them is the pivot)
  
%   $\Downarrow$
  
%   $X=\sum_{i=1}^{n-1}\sum_{j=i+1}^{n} X_{ij}$ where 

%   $X_{ij}=\left\{\begin{array}{ll}
%   1 & {\rm If} ~z_i~ {\rm compared}~{\rm to}~ z_i \\
%   0 & {\rm If} ~z_i~ {\rm not}~{\rm compared}~ \rm{to} ~z_i
% \end{array}
%   \right.$
  
%   $\Downarrow$
  
%   $\begin{array}{lcl}
%     E[X] &=& E\left[\sum_{i=1}^{n-1}\sum_{j=i+1}^{n} X_{ij}\right] \\
%     &=& \sum_{i=1}^{n-1}\sum_{j=i+1}^{n} E[X_{ij}] \\
%     &=& \sum_{i=1}^{n-1}\sum_{j=i+1}^{n} Pr[z_i ~{\rm compared}~{\rm
% 	to}~ z_j]
%   \end{array}$
  
% \item To compute $Pr[z_i ~{\rm compared}~{\rm to}~ z_j]$ it is useful to
%   consider when two elements are \emph{not} compared.
  
%   \begin{itemize}
%   \item[] Example: Consider an input consisting of numbers 1 through
%     $n$.
    
%     Assume first pivot it 7 $\Rightarrow$ first partition separates
%     the numbers into sets $\{1,2,3,4,5,6\}$ and $\{8,9,10\}$.
    
%     In partitioning, 7 is compared to all numbers. No number from the
%     first set will ever be compared to a number from the second set.
%   \end{itemize}
  
%   In general, once a pivot $x$, $z_i<x<z_j$, is chosen, we know
%   that $z_i$ and $z_j$ cannot later be compared.
  
%   On the other hand, if $z_i$ is chosen as pivot before any other
%   element in $Z_{ij}$ then it is compared to each element in
%   $Z_{ij}$. Similar for $z_j$.
  
%   \begin{itemize}
%   \item[] In example: 7 and 9 are compared because 7 is first item from
%     $Z_{7,9}$ to be chosen as pivot, and 2 and 9 are not compared because the
%     first pivot in $Z_{2,9}$ is 7.
%   \end{itemize}
  
%   Prior to an element in $Z_{ij}$ being chosen as pivot, the set
%   $Z_{ij}$ is together in the same partition $\Rightarrow$ any element
%   in $Z_{ij}$ is equally likely to be first element chosen as pivot
%   $\Rightarrow$ the probability that $z_i$ or $z_j$ is chosen first in
%   $Z_{ij}$ is $\frac{1}{j-i+1}$

%   $\Downarrow$

%   $Pr[z_i ~{\rm compared}~{\rm to}~ z_j]=\frac{2}{j-i+1}$

%   \item We now have:

%   $\begin{array}{lcl}
%   E[X] &=& \sum_{i=1}^{n-1}\sum_{j=i+1}^{n} Pr[z_i ~{\rm compared}~{\rm
%              to}~ z_j] \\
%        &=& \sum_{i=1}^{n-1}\sum_{j=i+1}^{n} \frac{2}{j-i+1} \\
%        &=& \sum_{i=1}^{n-1}\sum_{k=1}^{n-i} \frac{2}{k+1} \\
%        &<& \sum_{i=1}^{n-1}\sum_{k=1}^{n-i} \frac{2}{k} \\
%        &=& \sum_{i=1}^{n-1} O(\log n) \\
%        &=& O(n\log n)
%   \end{array}$  

% \item Since best case is $\theta(n \lg n)$ $\Longrightarrow  E[X] =
% \Theta(n \lg n)$ and therefore $E[T(n)] = \Theta(n \lg n)$. 

% \end{itemize}




\section*{NOTES}

\begin{itemize} 

\item In a future lectore (selection) we will see how to make
  quicksort run in worst-case $O(n\log n)$ time.


\item Trivia: Quicksort was invented by Sir Charles Antony Richard
  “Tony” Hoare, a British computer scientist and winner of the 1980
  Turing Award, in 1960, while he was a visiting student at Moscow
  State University.  While in Moscow he was in the school of
  Kolmogorov (well known mathematician).

\item The original Quickort was described with a different partition
  algorithm, one that works from both ends; this is referd to as
  Hoare-partition (after Hoare who invented Quicksort).  We'll look at
  this different way to partition in the lab.

\item Both Hoare's and Lomuto partitions are in-place, but none of
  them is stable.

\item Thus, in an efficient implementation, with an in-place
  partition, Quicksort is NOT stable.

\item I don't know of any practical comparison of the two partitions,
  and I would guess they are both equally fast in practice.
  Lomuto-partition may be considered simpler and better, because it
  uses a single pointer (i, to point to the last element left of
  pivot)---and this is why textbooks prefer it.  OTOH Hoare-partition
  does well (ie $O(n \lg n)$) when all elements are equal, while
  Lomuto-partition is quadratic.

\item The idea behind Lomuto's partition can be extended to deal with
  elements equal to the pivot (place them in the middle instead of
  carrying them through recursion); also with 3-way partitioning.

\item Quicksort can be made stable by using extra space (thus not in
  place anymore).

\end{itemize} 

\end{document}
