\documentclass[11pt]{article}

\usepackage{fullpage}
\usepackage{graphicx,epsfig}
\usepackage{color} 

\begin{document}

\title{{\bf Minimum Spanning Trees} \\
\normalsize{(CLRS 23)}}
\date{}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The problem}

Recall the definition of a \emph{spanning tree}: Given connected,
undirected graph $G=(V,E)$, a subset of edges of $G$ such that they
connect all vertices in $G$ and form no cycles is called a {\em
  spanning tree} (ST) of $G$.  Any undirected, connected graph has a
spanning tree. Actually, a graph usually has many spanning trees.

\vspace{.5cm}
Question: How would you go about finding a ST of a given graph $G$?
Answer: Run BFS or DFS; the resulting BFS- or DFS-tree are spanning
trees of $G$.
\vspace{.5cm}


The {\em minimum spanning tree} (MST) problem is the following: Given
a connected, undirected, weighted graph $G$ (each edge $(u,v)$ has
weight $w(u,v)$), find a spanning tree $T$ of minimum weight: $w(T) =
\sum_{(u,v) \in T} w(u,v)$. In other words, of all spanning trees of
$G$, we want one of minimum total weights. We use {\em minimum
  spanning tree} as short for {\em minimum weight spanning tree}).


\vspace{.5cm}
Example: \\
\centerline{\includegraphics[height=3.5cm]{mst_example.pdf}}
\begin{itemize}
\item Weight of MST is $4+8+7+9+2+4+1+2=37$
\item Note: MST is not unique: e.g. $(b,c)$ can be exchanged with
  $(a,h)$
\end{itemize}


The MST problem has many applications: For example, think about
connecting cities with minimal amount of wire or roads (cities are
vertices, weight of edges are distances between city pairs). Also,
network design in general (telephones, electrical, TV cable, road
networks).  


\newpage  
\section{MST algorithms}

The two classical algorithms for computing MST are Kruskal's and
Prim's algorithms.  They both have same running time complexity, and
they are both greedy algorithms, but they use complementary
approaches: 

\begin{itemize} 

\item Kruskal's algorithms: Start with a $T$ consisting of all
  vertices but no edges. Consider the edges in $G$ in increasing order
  of weight.  Add edge $e$ to $T$ unless doing so would create a
  cycle.

  This corresponds to growing the tree from a forest of $V$
  disconnected vertices.  Add one edge at a time, joining two trees in
  the forest together. Each join reduces the number of connected
  components in the forest by one.

\item Prim's algorithm: Start with an empyt $T$, and an arbitrary
  vertex $v$.  Grow $T$ one edge at a time: at each step, add to $T$
  the edges of minimum weight that has exactly one endpoint in $T$.

  This grows a tree one edge at a time. The current tree is always
  connected but only covers a part of the vertices.

\end{itemize} 


\section{Correctness}

The correctness of both MST algorithms follows from the following
general theorem:

\vspace{.5cm}
{\bf The cut theorem:} Let $S$ be a set of vertices in a graph
$G=(V,E)$, and consider all edges with one endpoint in $S$ and the
other one in $V-S$.\footnote{The partition of $V$ into $(S, V-S)$ is
  referred to as a \emph{cut}, and the edges with one one endpoint in
  $S$ and the other one in $V-S$ are called edges that \emph{cross the
    cut}.}  Let $e=(u,v)$ be an edge of minimum weight among all edges
that cross the cut. Then there exists a MST that contains $e$.

\vspace{.5cm}
Proof: 
 %    Example: Cut $S$ respects $T$ \\
%     \centerline{\includegraphics[height=5cm]{cut_example.pdf}}
%   \end{itemize}
% {\em Theorem}: If $G=(V,E)$ is a graph such that $T \subseteq
%   E$ is subset of some MST of $G$, and $S$ is a cut respecting $T$ {\bf
%     then} there is a MST for $G$ containing $T$ and the minimum weight edge
%   $e=(u,v)$ crossing $S$.
Let $T$ be a MST of $G$.  If $T$ contains edge $e=(u,v)$, then we are
done.  If $T$ does not contain $e=(u,v)$, then: 
\begin{itemize} 
\item  Because $T$ is a tree, there is a unique path in $T$ between any two
nodes and in particular between $u,v$. 
\item Because $u$ is on one side of
the cut and $v$ is on the other, this path must contain at least an
edge $(x,y) \in T$ crossing the cut.
\item  When adding $e=(u,v)$ to $T$, this path and $e$ form a cycle.
\item If we remove edge $(x,y)$ from $T$ and add $e$ instead, then:
\item $T' = T -(x,y) + (u,v)$ is a spanning tree; and 
  
\item $T'$ must have same weight as $T$
  since $w(u,v) \leq w(x,y)$ \\
  $\Downarrow$ \\
  Thus $T'$ is also a MST, and thus there is a MST containing $T$
  and $e$.
\end{itemize}


The Cut Theorem allows us to describe a very abstract greedy algorithm
for MST:

\vspace{.5cm}
\fbox{
  \parbox{8cm}{
    $T = \emptyset$ \\
    While $\vert T \vert \leq \vert V \vert - 1$ DO
    \begin{itemize}
    \item[] Find cut $S$ respecting $T$
    \item[] Find minimal edge $e$ crossing $S$
    \item[] $T = T \cup \{ e \}$
    \end{itemize}
  }} 

\vspace{.5cm}

Both Prim's and Kruskal's algorithms follow this abstract algorithm
and their correctness follows from the cut theorem by chosing a
suitable cut:

\begin{itemize} 

\item Prim's: The cut is $(T, V-T)$, that is, the current tree on one
  side, and the remaining vertices on the other. At each point the
  algorithm adds the minimum edge crossing the cut, which, by the cut
  theorem, we know must be part of an MST.

\item Kruskal's: When adding edge ($u,v)$ chose a cut that has the
  vertices in the set of $u$ on one side and the vertices in the set
  of $v$ on the other. Edge $e$ is the smallest edge that crosses the
  cut and thus by the cut theorem, we know must be part of an MST.

\end{itemize} 






\section{PRIM's algorithm}

GeneraL IDEA: 
\begin{itemize}
\item Start with spanning tree containing arbitrary vertex $r$ and
  no edges
\item Grow spanning tree by repeatedly adding minimal weight edge
  connecting vertex in current spanning tree with a vertex not in the
  tree
\end{itemize}


Implementation:
\begin{itemize}
\item To find minimal edge connected to current tree we maintain a
  priority queue on vertices not in the tree. The key/priority of a
  vertex $v$ is the weight of minimal weight edge connecting $v$ to
  the tree. (We maintain pointer from adjacency list of $v$ to $v$ in
  the priority queue).
\item For each node $v$ maintain $visit(v)$ such that edge $(v,
  visit(v))$ is the best edge connecting $v$ to the current tree.
\end{itemize}

\fbox{
  \parbox{10cm}{
    PRIM(r)\\   
    \textcolor{blue}{/* initialize */ \\}
    For each vertex  $u \in V, u \neq r$ DO: {\sc Insert}($PQ,u,\infty$)\\    
    {\sc Insert}$(PQ,r,0)$,  $visit(r) = NULL$ \\
    \textcolor{blue}{/* main loop */}\\
    WHILE $PQ$ not empty DO
    \begin{itemize}
    \item[] $u$ = {\sc Delete-min}($PQ$)
    \item[] For each $(u,v) \in E$ DO
      \begin{itemize}
      \item[] IF $v \in PQ$ and $w(u,v) < $ key($v$): 
      \item[] ~~~~visit[$v$] = $u$
      \item[] ~~~~{\sc Decrease-Key}($PQ,v,w(u,v)$)
      \end{itemize}
    \end{itemize}
    Output edges $(u, visit(u))$ as part of MST.
   
}
}

\vspace{.5cm}
On the example graph, the greedy algorithm would work as follows
(starting at vertex $a$): \\

\centerline{\includegraphics[height=6cm]{mst_prim_1.pdf}}
\centerline{\includegraphics[height=6cm]{mst_prim_2.pdf}}

\newpage
{\bf Prim's analysis: }
\begin{itemize}
\item $\vert V \vert$  {\sc INSERT}'s
\item While loop runs $\vert V \vert$ times $\Rightarrow$ we
  perform $\vert V \vert$ {\sc Deletemin}'s
\item We perform at most one {\sc Decrease-Key} for each of the $\vert E
  \vert$ edges 
\item If use a heap as pqueue, inserts, delete-min and decrease-key
  run in $O(\lg V) \Longrightarrow$ $O((\vert V \vert + \vert E \vert)
  \log \vert V \vert)=O(\vert E \vert \log \vert V \vert)$ 
  time.
\end{itemize}





\section{Kruskal's Algorithm}

General idea: 
\begin{itemize}
\item Start with a forest consisting of $|V|$ trees, each one consisting of one vertex
\item Consider edges $E$ in increasing order of their weight; add  an edge
  if it connects two trees, ie if it does not create cycles.
\end{itemize}

Example: \\ \\
\centerline{\includegraphics[height=6cm]{mst_kruskal_1.pdf}} \\ \\
\centerline{\includegraphics[height=6cm]{mst_kruskal_2.pdf}} \\


To implement Kruskal's idea, we need to a structure that maintains the
``partial'' trees and is able to support the following operations: (a)
initialize a tree as consisting of a single vertex; (b) are two
vertices in the same tree?; and (c) join two trees together. 

Abstracting further, we can think of each tree as a set of vertices.
We need a data structure to store each set of vertices so that we are
able to: (a) create a set consisting of one vertex; (b) determine of
two vertices are in the same set; (c) join two sets.

This type of structure is referred to as an \emph{union-find data
  structure} and its operations are referred to as: 
\begin{itemize}
\item {\sc Make-set}($v$): Create set consisting of $v$
\item {\sc Union-set}($u,v$): Unite set containing $u$ and set
  containing $v$
\item {\sc Find-set}($u$): Return unique representative for
  set containing $u$
\end{itemize}

We can re-write Kruskal's algorithm to use a union-find data structure
as follows:

\vspace{.5cm}
\fbox{
  \parbox{10cm}{
    KRUSKAL \\
    
    $T = \emptyset$ \\
    FOR each vertex $v \in V$: {\sc Make-Set}($v$)
    
    Sort edges of $E$ in increasing order by weight \\
    FOR each edge $e=(u,v) \in E$ in order DO
    \begin{itemize}
    \item[] IF {\sc Find-Set}($u$) $\neq$ {\sc Find-Set}($v$) THEN
      \begin{itemize}
      \item[] $T = T \cup \{ e \}$
      \item[] {\sc Union-Set}($u,v$)
      \end{itemize}
    \end{itemize}
  }}

\vspace{.5cm}
{\bf Kruskal's analysis: }
\begin{itemize}
\item We use $O(|E|\log |E|) = O(|E|\log |V|)$ time to sort edges and we perform
  $\vert V \vert$ {\sc Make-Set}, $\vert V \vert -1$ {\sc Union-set},
  and $2 \vert E \vert$ {\sc Find-Set} operations.
\item We will discuss a simple solution to the \emph{Union-Find
    problem} such that {\sc Make-Set} and {\sc Find-Set} take $O(1)$
  time and {\sc Union-Set} takes $O(\log V)$ time amortized.  With
  this implementation Kruskal's algorithm runs in time $O(|E|\log |E|
  + |V|\log |V|)=O((|E|+|V|)\log |E|) = O(|E|\log |V|)$ like Prim's
  algorithm.

\end{itemize}




\newpage
\section{Union-Find}


Kruskal's MST algorithm can be expressed in terms of an union-find
data structure.  As it turns out this is a general-purpose structure
that has other applications. Generally speaking the \emph{Union-Find
  problem} is the following: We want to provide a data structure to
maintain a set of elements under the following operations:
  \begin{itemize}
  \item {\sc Make-set}($v$): Create set consisting of $v$
  \item {\sc Union-set}($u,v$): Unite set containing $u$ and set
    containing $v$
  \item {\sc Find-set}($u$): Return unique representative for
    set containing $u$
  \end{itemize}

 
  \subsection{Simple solution} 

  Perhaps the simplest idea that comes to mind is to maintain elements
  in same set as a linked list with each element having a pointer to
  the first element in the list (unique
  representative).     Example: \\
    
    \centerline{\includegraphics[height=5cm]{union-find_linked.pdf}}
    
    \begin{itemize} 
    \item {\sc Make-Set}($v$): Make a list with one element
      $\Rightarrow O(1)$ time
    \item {\sc Find-Set}($u$): Follow pointer and return unique
      representative $\Rightarrow O(1)$ time
    \item {\sc Union-Set}($u,v$): Link first element in list with
      unique representative {\sc Find-Set}($u$) after last element
      in list with unique representative {\sc Find-set}($v$) $\Rightarrow
      O(|V|)$ time (as we have to update all unique representative
    pointers in list containing $u$)
  \end{itemize} 

  With this simple solution the $\vert V \vert -1$ {\sc Union-Set}
  operations in Kruskal's algorithm may take $O(\vert V \vert ^2)$ time.
  

  
  \subsection{Improved solution:} 

  We can improve the performance of {\sc Union-Set} with a very simple
  modification:

\vspace{.5cm}
  \fbox{
    \parbox{15cm}{ Always link the smaller list after the longer list
      (update the pointers of the smaller list) 
}}

\vspace{.5cm} If you think: this is simple, you are right!  As we'll
show below, such a beautifully simple and intuitive idea results in an
amortized bound of $O(\log |V|)$ per {\sc Union-Set}, amortized.  This
is algorithmic elegance at its best!

Why is this efficient? 
\begin{itemize}
\item One {\sc Union} operation takes time proportional to number of
  pointer changes it needs to do


\item One {\sc Union-Set} operation can still take $O(\vert V \vert)$
  time (It is possible that we join a list of V/2 vertices to a list
  of V/2 vertices, which requires V/2 pointer updates).

\item Kruskal's algorithm performs $\vert V \vert -1$ {\sc Union}
  operations in total

\item The trick is to think \emph{over all the {\sc Union} operations
    performed by Kruskal's algorithm}.




\item Consider element $u$:
    
  After pointer for $u$ is updated, $u$ belongs to a list of
  size at least double the size of the list it was in before the union\\
  $\Downarrow$ \\
  After $k$ pointer changes, $u$ is in list of size at least
  $2^k$ \\
  $\Downarrow$ \\
  Since the size of the largest set is $|V|$, the pointer of $u$ can
  be changed at most $\log \vert V \vert$ times.
  
\item In other words, over the course of all the {\sc Union}
  operations, each element can have its pointer changed only $O(\lg
  V)$ times.

\item This means that $O(V)$ {\sc Union} operations will take $O(\vert
  V \vert \log \vert V \vert)$ time altogether in the worst case

% \item one {\sc Union-Set}
%   takes $O(\log \vert V \vert)$ time \emph{amortized}):

\end{itemize}

Notes: With improvement, Kruskal's algorithm runs in time $O(|E|\log
|E| + |V|\log |V|)=O((|E|+|V|)\log |E|) = O(|E|\log |V|)$ like Prim's
algorithm.



\section{Sate-of-the-art} 

A Union-Find data structure with better bounds is possible; however
even when using an improved UF structure Kruskal's algorithm stays at
$O(|E|\log |V|)$ and its running time will be dominated by sorting.

The best upper bound for computing MST is
$O(|E| \alpha(|E|, |V|))$ where $\alpha()$ is the inverse Ackerman
function; $\alpha()$ is a very slowly growing function, in practice a
constant. MST can be computed in expected $O(V+E)$ time using a
randomized algorithm---the algorithm was a breakthrough when first
presented by Tarjan etal in 1995.  Computing MST in deterministic
linear time (using comparisons) is a big open problem.


\end{document} 
