Clustering and dimensionality reduction

What you will do: find groups in data without being told what the groups are (clustering), choose how many groups to use, describe them in plain words, and then squash many columns into two so you can draw them (dimensionality reduction with PCA). You will use the classic iris flower data and the handwritten digits data, both built into scikit-learn.

Prerequisites: pandas basics, and *How machine learning learns: supervised vs unsupervised*. Run everything in Colab or Kaggle.

In supervised learning every row has a label. In unsupervised learning there are no labels; the algorithm looks for structure by itself. Two common jobs:

- Clustering: group similar rows together. Example: segment shoppers into groups for different marketing. - Dimensionality reduction: represent many columns with fewer, keeping as much information as possible. Example: plot a 64-column dataset on a 2D chart, or remove redundant columns before modelling.

K-means is the most common clustering method. You choose k, the number of clusters. Then:

1. Place k points (centroids) at random. 2. Assign every row to its nearest centroid. 3. Move each centroid to the average position of its rows. 4. Repeat steps 2 and 3 until nothing changes.

It measures distance between rows, so, just like KNN, you must scale your features. A column in thousands (salary) would otherwise decide everything. It also assumes clusters are roughly round blobs of similar size, and the result depends on the random start, so we run it several times (n_init) and keep the best.

import pandas as pd
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

iris = load_iris(as_frame=True)
X = iris.data                        # 150 flowers, 4 measurements. We will NOT use iris.target.
X_scaled = StandardScaler().fit_transform(X)
print(X.head(3))

The dataset has 150 flowers of 3 species. We keep only the 4 measurements, as if we did not know the species.

There is usually no single right k. Two helpful measures:

- Inertia: the total squared distance from each row to its centroid. It always falls as k grows, so look for the elbow, where the improvement slows down. - Silhouette score: from -1 to 1, how well each row fits its own cluster compared with the next nearest one. Higher is better. It can go down as k grows, which makes it easier to read.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=42)
    labels = km.fit_predict(X_scaled)
    print(k, "inertia:", round(km.inertia_, 1), "silhouette:", round(silhouette_score(X_scaled, labels), 3))

n_init=10 runs 10 different random starts for each k.

Silhouette is highest at k = 2, and inertia has a bend near k = 3. Both are reasonable answers. Two clusters is the cleanest split: one group of flowers that are clearly separate from the rest. We know there are really three species, but two of them overlap, so the data alone only slightly supports three groups. This is a normal outcome: the numbers guide you, but the final choice depends on what is useful for the business question.

A cluster is only useful if you can say what it is. Profile each one by averaging the original (unscaled) columns per cluster. Then, only because iris happens to have known species, compare with them.

kmeans = KMeans(n_clusters=3, n_init=10, random_state=42)
clusters = kmeans.fit_predict(X_scaled)

# Profile each cluster in the original units
pd.set_option("display.width", 120)
pd.set_option("display.max_columns", 20)
profile = X.copy()
profile["cluster"] = clusters
print(profile.groupby("cluster").mean().round(2))
print(profile["cluster"].value_counts().sort_index())

# Only because iris happens to have known species can we check our clusters
print(pd.crosstab(clusters, iris.target_names[iris.target]))

Cluster numbers are arbitrary: cluster 0 in your run may be cluster 2 in someone else's.

Cluster 1 is exactly the setosa species: small flowers, found perfectly. Clusters 0 and 2 mix versicolor and virginica, because those species overlap. With real business data you will not have the species column; you would name the clusters by their profile, for example "high-spend, frequent visitors" and "occasional, low-spend".

The exact output may differ slightly with other library versions.

Principal component analysis (PCA) finds new axes, called components, that are combinations of the original columns. The first component is the direction along which the data varies most. The second is the best direction at right angles to the first, and so on. If you keep only the first two, you can plot the data on a flat chart with minimal loss.

Each component has an explained variance ratio: the share of the total spread it keeps. PCA also needs scaled input, otherwise large-number columns dominate.

import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

pca = PCA(n_components=2, random_state=42)
X_2d = pca.fit_transform(X_scaled)
print("Variance kept by each component:", pca.explained_variance_ratio_.round(3))

plt.figure(figsize=(6, 4))
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=clusters)
plt.xlabel("Component 1")
plt.ylabel("Component 2")
plt.title("Iris clusters in 2D (PCA)")
plt.tight_layout()
plt.savefig("iris_clusters.png", dpi=120)   # in Colab or Jupyter you can use plt.show() instead

Reduces 4 columns to 2 and colours the points by cluster. Saves a PNG so it also works in a plain script.

Two components keep about 95.9% (0.730 + 0.229) of the information in the four iris columns, so the 2D plot is faithful. The saved chart shows one clearly separate group and two touching ones, matching what we found.

For data with many columns, plot or print the cumulative explained variance and keep enough components to reach a target such as 95%. The handwritten digits dataset has 64 columns (8 x 8 pixels per image).

from sklearn.datasets import load_digits

digits = load_digits()
print(digits.data.shape)               # 1797 images, 64 pixels each (8 x 8)

pca_full = PCA().fit(digits.data)
cumulative = pca_full.explained_variance_ratio_.cumsum()
n_95 = int((cumulative < 0.95).sum()) + 1
print("Components needed to keep 95% of the variance:", n_95)

Run after the previous cells (PCA is already imported).

Twenty-nine components hold 95% of the variance of 64 pixels. That is a useful way to shrink data, remove noise and speed up later models.

PCA is linear and its components are mixtures of columns, so they are harder to explain than the original columns. For pure visualisation of complex data, t-SNE and UMAP often give nicer pictures; use them for looking, not as a basis for conclusions about distances.

Typical questions: "How does k-means work?", "How do you choose k?" (elbow, silhouette, domain need), "Why scale before k-means?", "What are the limits of k-means?" (round clusters, sensitive to start and outliers, needs k) and "What does PCA do and when would you use it?". Another favourite: how would you segment customers and how would you check the segments are useful.

I clustered [what the rows are, e.g. customers of an online shop] with k-means (k = [number]) using these features: [feature list]. Here are the average values per cluster:

[paste the profile table with cluster sizes]

Give each cluster a short, plain-English name and a one-sentence description. Then suggest one action a business could take for each cluster, and tell me what extra checks I should run before trusting these segments.

What to expect: A good answer bases names on the numbers, not on guesses, and suggests checks such as stability across seeds and validation with business experts.