Hierarchical clustering
In data mining and statistics, hierarchical clustering is a method of cluster analysis that seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two categories:
- Agglomerative: Agglomerative clustering, often referred to as a "bottom-up" approach, begins with each data point as an individual cluster. At each step, the algorithm merges the two most similar clusters based on a chosen distance metric and linkage criterion. This process continues until all data points are combined into a single cluster or a stopping criterion is met. Agglomerative methods are more commonly used due to their simplicity and computational efficiency for small to medium-sized datasets.
- Divisive: Divisive clustering, known as a "top-down" approach, starts with all data points in a single cluster and recursively splits the cluster into smaller ones. At each step, the algorithm selects a cluster and divides it into two or more subsets, often using a criterion such as maximizing the distance between resulting clusters. Divisive methods are less common but can be useful when the goal is to identify large, distinct clusters first.
Hierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a matrix of distances. On the other hand, except for the special case of single-linkage distance, none of the algorithms can be guaranteed to find the optimum solution.
Complexity
The standard algorithm for hierarchical agglomerative clustering has a time complexity of and requires memory, which makes it too slow for even medium data sets. However, for some special cases, optimal efficient agglomerative methods are known: SLINK for single-linkage and CLINK for complete-linkage clustering. With a heap, the runtime of the general case can be reduced to instead of, at the cost of additional the memory requirements. In many cases, the memory overheads of this approach are too large to make it practically usable. Methods exist which use quadtrees that demonstrate total running time with space.Divisive clustering with an exhaustive search is, but it is common to use faster heuristics to choose splits, such as k-means.
Cluster linkage
In order to decide which clusters should be combined, or where a cluster should be split, a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by use of an appropriate distance d, such as the Euclidean distance, between single observations of the data set, and a linkage criterion, which specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets. The choice of metric as well as linkage can have a major impact on the result of the clustering, where the lower level metric determines which objects are most similar, whereas the linkage criterion influences the shape of the clusters. For example, complete-linkage tends to produce more spherical clusters than single-linkage.The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.
Some commonly used linkage criteria between two sets of observations A and B and a distance d are:
| Names | Formula |
| Maximum or complete-linkage clustering | |
| Minimum or single-linkage clustering | |
| Unweighted average linkage clustering | |
| Weighted average linkage clustering | |
| Centroid linkage clustering, or UPGMC | where and are the centroids of A resp. B. |
| Median linkage clustering, or WPGMC | where |
| Versatile linkage clustering | |
| Ward linkage, Minimum Increase of Sum of Squares | |
| Minimum Error Sum of Squares | |
| Minimum Increase in Variance | |
| Minimum Variance | |
| Hausdorff linkage | |
| Minimum Sum Medoid linkage | such that m is the medoid of the resulting cluster |
| Minimum Sum Increase Medoid linkage | |
| Medoid linkage | where, are the medoids of the previous clusters |
| Minimum energy clustering |
Some of these can only be recomputed recursively, for many a recursive computation with Lance-Williams-equations is more efficient, while for other the distances have to be computed with the slower full formula. Other linkage criteria include:
- The probability that candidate clusters spawn from the same distribution function.
- The product of in-degree and out-degree on a k-nearest-neighbour graph.
- The increment of some cluster descriptor after merging two clusters.
Agglomerative clustering example
The hierarchical clustering dendrogram would be:
Image:Hierarchical clustering simple diagram.svg|frame|none|Traditional representation
Cutting the tree at a given height will give a partitioning clustering at a selected precision. In this example, cutting after the second row of the dendrogram will yield clusters . Cutting after the third row will yield clusters , which is a coarser clustering, with a smaller number but larger clusters.
This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements and. The first step is to determine which elements to merge in a cluster. Usually, we want to take the two closest elements, according to the chosen distance.
Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described in the single-linkage clustering page; it can easily be adapted to different types of linkage.
Suppose we have merged the two closest elements b and c, we now have the following clusters,,, and, and want to merge them further. To do that, we need to take the distance between and, and therefore define the distance between two clusters.
Usually the distance between two clusters and is one of the following:
- The maximum distance between elements of each cluster :
- The minimum distance between elements of each cluster :
- The mean distance between elements of each cluster :
- The sum of all intra-cluster variance.
- The increase in variance for the cluster being merged
- The probability that candidate clusters spawn from the same distribution function.
One can always decide to stop clustering when there is a sufficiently small number of clusters. Some linkages may also guarantee that agglomeration occurs at a greater distance between clusters than the previous agglomeration, and then one can stop clustering when the clusters are too far apart to be merged. However, this is not the case of, e.g., the centroid linkage where the so-called reversals may occur.
Divisive clustering
The basic principle of divisive clustering was published as the DIANA algorithm. Initially, all data is in the same cluster, and the largest cluster is split until every object is separate.Because there exist ways of splitting each cluster, heuristics are needed. DIANA chooses the object with the maximum average dissimilarity and then moves all objects to this cluster that are more similar to the new cluster than to the remainder.
Informally, DIANA is not so much a process of "dividing" as it is of "hollowing out": each iteration, an existing cluster is chosen to form a new cluster inside of it. Objects progressively move to this nested cluster, and hollow out the existing cluster. Eventually, all that's left inside a cluster is nested clusters that grew there, without it owning any loose objects by itself.
Formally, DIANA operates in the following steps:
- Let be the set of all object indices and the set of all formed clusters so far.
- Iterate the following until :
- # Find the current cluster with 2 or more objects that has the largest diameter:
- # Find the object in this cluster with the most dissimilarity to the rest of the cluster:
- # Pop from its old cluster and put it into a new splinter group.
- # As long as isn't empty, keep migrating objects from to add them to. To choose which objects to migrate, don't just consider dissimilarity to, but also adjust for dissimilarity to the splinter group: let where we define, then either stop iterating when, or migrate.
- # Add to.
The dendrogram of DIANA can be constructed by letting the splinter group be a child of the hollowed-out cluster each time. This constructs a tree with as its root and unique single-object clusters as its leaves.