Principal Component Analysis (PCA)

1. Standardise: Centre and scale features to zero mean and unit variance (StandardScaler). Critical — PCA is scale-sensitive.
2. Compute covariance matrix: Σ = (1/n) XᵀX (X already mean-centred). Σ is (p×p) symmetric positive semi-definite.
3. Eigendecomposition: Decompose Σ = VΛVᵀ where columns of V are eigenvectors (principal components) and Λ is diagonal with eigenvalues λ₁ ≥ λ₂ ≥ ... ≥ λₚ.
4. Select top-K eigenvectors: Choose K eigenvectors with largest eigenvalues — they explain the most variance.
5. Project: Transform data onto the K principal components: Z = XV_K (where V_K is the p×K matrix of top-K eigenvectors).

The proportion of total variance explained by each component. Sum of all ratios = 1 (100%). Choose K so cumulative variance ≥ 95% (or 99%). This is how GATE tests PCA: given eigenvalues, compute explained variance and choose components.

PCA from scratch using eigendecomposition + sklearn

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data     # 150×4 (sepal length, sepal width, petal length, petal width)

# ── PCA from scratch ──
# Step 1: Standardise
X_scaled = (X - X.mean(axis=0)) / X.std(axis=0)

# Step 2: Covariance matrix
Sigma = np.cov(X_scaled, rowvar=False)  # (4×4)
print("Covariance matrix shape:", Sigma.shape)

# Step 3: Eigendecomposition
eigenvalues, eigenvectors = np.linalg.eigh(Sigma)
# Sort descending (eigh returns ascending)
idx = np.argsort(eigenvalues)[::-1]
eigenvalues  = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]

print("Eigenvalues:", eigenvalues.round(3))
print("Explained variance ratio:", (eigenvalues/eigenvalues.sum()).round(3))
print("Cumulative explained variance:", np.cumsum(eigenvalues/eigenvalues.sum()).round(3))

# Step 4: Project onto top-2 PCs
V2 = eigenvectors[:, :2]        # (4×2)
Z  = X_scaled @ V2              # (150×2)
print(f"Original shape: {X.shape} → Reduced: {Z.shape}")

# ── sklearn PCA ──
pca = PCA(n_components=2)
scaler = StandardScaler()
X_pca = pca.fit_transform(scaler.fit_transform(X))
print(f"sklearn explained variance: {pca.explained_variance_ratio_.round(3)}")
# PC1 explains ~73%, PC2 explains ~23% → 2 PCs capture ~96% of variance!

1. A dataset has 10 features. Eigenvalues of the covariance matrix are [5, 3, 2, 1, 0.5, 0.3, 0.1, 0.1, 0.05, 0.05]. How many PCs capture ≥ 90% variance? (Answer: Total = 12.1. Cumulative: 5(41%), 8(66%), 10(82%), 11(91%). Three PCs capture 83%, four capture 91% → need 4 PCs for ≥90%.)
2. Why must data be standardised before PCA? (Answer: PCA maximises variance. Features with large measurement scales (e.g., salary in thousands vs. age in years) dominate the covariance matrix. Standardising ensures all features contribute equally.)
3. Principal components are always: (Answer: Orthogonal to each other (perpendicular in feature space) — they are uncorrelated by construction. This is the key property that makes PCA useful for removing correlation.)
4. If all features are already uncorrelated, what does PCA produce? (Answer: Principal components aligned with the original feature axes — PCA finds nothing new. The eigenvectors are the standard basis vectors. PCA is most useful when features are correlated.)
5. PCA reduces a 1000-feature dataset to 10 PCs. A new test point has 1000 features. How do you project it? (Answer: Subtract the training mean, divide by training std (same scaler), then multiply by the 1000×10 matrix of top-10 eigenvectors. Never refit the scaler or PCA on test data.)