Phase 2: Classical Machine Learning

K-Means, Hierarchical & DBSCAN Clustering

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a giant pile of all your toys on the floor – cars, LEGOs, action figures, stuffed animals, art supplies, everything! It's a bit messy, right? It's much easier to find what you want if things are organized into groups. That's exactly what "clustering" helps computers do: take a big jumble of information and sort it into meaningful groups automatically.

Let's say you want to sort your toys into three groups, maybe for three different shelves. With one method, called K-Means, you'd first pick three random toys from the pile – let's say a toy car, a LEGO brick, and a small stuffed animal. These are like your starting "leaders" for each group. Now, you'd look at every single toy in your pile and decide: "Is this toy more like the car, the LEGO brick, or the stuffed animal?" You'd put it with the leader it's most similar to.

Once all toys are sorted, you'd then look at all the toys in the "car" group and find the actual average toy that best represents that whole group – maybe a slightly bigger truck. You'd do the same for the LEGO and stuffed animal groups, moving your "leaders" to be the true center of their new groups. You'd keep doing this: sort all toys to their new closest leader, then move the leaders again. Eventually, the leaders stop moving because they're perfectly centered in their groups. This is super handy for sorting things when you already know how many groups you want, like if you know you have exactly three shelves for your toys.

But what if you don't know how many shelves you have, or you want to see how different toys relate to each other? Another way, called Hierarchical Clustering, works like building a family tree for your toys. You start by treating every single toy as its own tiny group. Then, you find the two toys that are most similar to each other – maybe two identical LEGO bricks – and you stick them together to make a small pair-group. You then find the next two closest things, maybe a toy car and a toy truck, and group them. You keep doing this, merging the two most similar groups (not just single toys anymore) together, until eventually, all your toys are in one giant group. It's like you've built a pyramid where the bottom is every individual toy, and the top is the "all toys" group.

The cool part about building that toy-tree is you can then look at it and decide, "Hmm, I think I want three groups of toys," and you just "cut" the tree at a certain level. Or maybe you want six groups, so you cut it lower down. This means you don't have to guess how many groups you need upfront; you can see the relationships and choose later! So, when computers use these methods, they can automatically sort huge amounts of information, whether it's figuring out different types of customers for a store, or grouping similar animals in a giant database, helping us make sense of really big collections of things.

K-Means is a foundational partitioning algorithm that aims to divide your dataset into k distinct, non-overlapping clusters. The core idea is iterative: you start by randomly placing k centroids, then assign each data point to its closest centroid, and finally update each centroid to be the mean of all points assigned to it. This process repeats until the centroids no longer move significantly. It's computationally efficient and great for finding spherical-shaped clusters of similar sizes, making it popular for tasks like customer segmentation or initial data exploration. However, you must pre-specify the number of clusters k, and its performance can degrade with non-spherical clusters or varying densities.

Unlike K-Means, Hierarchical Clustering builds a tree-like structure of clusters called a dendrogram, allowing you to choose the number of clusters retrospectively. Agglomerative (bottom-up) is the most common approach: it starts by treating each data point as its own cluster, then iteratively merges the two closest clusters until all points belong to a single cluster. The "closeness" between clusters is defined by various linkage criteria (e.g., Ward, average, complete). This method is powerful for understanding the relationships and hierarchy within your data and doesn't require pre-defining k. However, it can be computationally more expensive for very large datasets and interpreting the optimal cut from a dendrogram can sometimes be subjective.

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) offers a fundamentally different approach, focusing on the density of data points. It identifies "core points" (points with many neighbors within a specified radius eps), "border points" (neighbors of core points but not dense enough themselves), and "noise points" (outliers that don't belong to any cluster). Clusters are formed by connecting all density-reachable core points and their border points. DBSCAN excels at discovering arbitrarily shaped clusters and is robust to noise, as it explicitly labels outliers. It doesn't require you to specify k, but its performance is highly sensitive to the eps (radius) and min_samples (minimum neighbors) parameters, making it challenging to tune for datasets with varying densities. Choosing the right algorithm depends heavily on your data's characteristics and the problem you're trying to solve.

Key Takeaways

  • K-Means is fast and good for spherical clusters but requires k upfront and is sensitive to initialization.
  • Hierarchical Clustering builds a dendrogram, allowing flexible k selection, but is computationally intensive for large datasets.
  • DBSCAN identifies arbitrary shapes and outliers (noise) based on density, but requires careful tuning of eps and min_samples.
  • Each algorithm suits different data structures and problem goals; understand their strengths and weaknesses to choose effectively.

Code Example

python
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

# 1. Generate sample data
X, _ = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)

# 2. Apply K-Means
# Instantiate K-Means with 4 clusters (assuming we know k)
kmeans = KMeans(n_clusters=4, random_state=0, n_init=10)
kmeans.fit(X)

# 3. Get cluster labels and centroids
labels = kmeans.labels_
centroids = kmeans.cluster_centers_

print("First 10 cluster labels:", labels[:10])
print("Cluster centroids:\n", centroids)

How this code works

This code demonstrates how to apply the K-Means clustering algorithm, a fundamental technique for grouping similar data points, to a synthetic dataset. It begins by creating sample data using make_blobs, which generates 300 data points organized into 4 distinct clusters, simulating a common scenario for unsupervised learning. The _ in X, _ indicates that while make_blobs can provide the "true" cluster assignments, K-Means is unsupervised, so we intentionally ignore them to learn the groupings from the data itself.

Next, the KMeans algorithm is initialized with n_clusters=4, telling it to look for exactly four groups, aligning with how the data was generated. A subtle but important detail is n_init=10. Because K-Means starts by randomly guessing initial cluster centers, running it multiple times with different starting points helps ensure the algorithm finds a more robust and optimal clustering, rather than getting stuck in a less ideal solution from a single attempt. The kmeans.fit(X) step then executes the clustering process. Finally, the labels_ attribute provides the assigned cluster for each data point, and cluster_centers_ gives the coordinates of the final centroids, revealing where each learned cluster is located.

Related Articles

Hand-picked reads to go deeper on unsupervised learning.

Back to ML Engineer Roadmap