Awesome Search KG

PCA

11 min read

PCA (Principal Component Analysis)

Definition

PCA is a linear dimensionality reduction technique that projects data onto a new coordinate system whose axes (principal components) are ordered by the amount of variance they explain. The first principal component captures the most variance; each subsequent one captures the next most, orthogonal to all previous.

How It Works

  1. Standardize — subtract mean, divide by std per feature
  2. Covariance matrix — n×n matrix summarizing pairwise feature correlations
  3. Eigendecomposition — eigenvectors = principal component directions; eigenvalues = variance explained
  4. Sort eigenvectors by eigenvalue descending
  5. Project — multiply original data by the top-k eigenvectors matrix → reduced representation

Properties

PropertyValue
TypeLinear
PreservesGlobal variance structure
SpeedFast (one-pass, deterministic)
ParametricYes — reusable on new data
Inverse transformYes (lossy)
Suitable for MLYes
  • Can reduce embedding dimensions (e.g., 768→256) before ANN indexing, cutting memory and speeding up search with modest quality loss
  • Useful when embedding dimensions have near-zero variance (exploited in HNSW with PCA preprocessing)
  • Alternative to Vector Quantization: DR reduces the number of dimensions; quantization reduces bits per dimension — both are often combined

Fitting: Do You Need All the Vectors in Memory?

The worry is well founded for the naive path: ten million 1,024-dim float32 vectors is ~41 GB, and sklearn.decomposition.PCA does want its input array materialized.

But what PCA is estimating is a d × d object — a covariance matrix, or equivalently the top-k right singular vectors — and that does not grow with the corpus. 1024² float64 is 8 MB whether you have ten thousand vectors or ten million. Corpus size is therefore a property of how you fit, not of what PCA fundamentally needs: an implementation that accumulates covariance over batches only has to stream the data, while sklearn’s default SVD paths decompose whatever array you hand them. The two practical routes below exploit this from opposite ends — one shrinks the input, the other streams it.

Three ways to fit, in rough order of how often they are the right call:

1. Representative sample (usually enough)

Fit on a random sample and apply the resulting projection to everything. The eigenvector directions stabilize long before the corpus is exhausted — you are estimating a d × d covariance, and the measurements below found much smaller samples sufficient than intuition suggests.

Two independent measurements bracket how small “enough” is: Doug Turnbull fit on 100K vectors of a 9M-document corpus (Principal Component Analysis - an embedding shrink-ray), and Dylan Castillo found a projection fit on 1,000 documents performed almost identically to one fit on the full corpus, and that a projection fit on a different corpus (MS MARCO) transferred usefully to unrelated datasets — down to 64 dims on a 1,536-dim model and 128 on a 4,096-dim one (Honey, I Shrunk the Embeddings - Matryoshka vs PCA).

A third data point scales the sample with dimensionality rather than corpus size: ASH fits its learned projection on ten times the embedding dimension — roughly 15K vectors for a 1,536-dim model — whether the corpus holds 100K vectors or a million. That is the shape the theory predicts, since what is being estimated grows with dimensions and not with documents, and it is a more transferable rule of thumb than any flat number.

Sample randomly, not by ingestion order — a prefix of a catalog is often one category.

What makes a sample “reasonable”

“Sample randomly, 10–100K” leaves the operative word undefined. The working criterion: the sample is large and representative enough that the principal components stop moving when you add more data — stability of the projection, not of any downstream score.

A conservative starting range for high-dimensional catalog embeddings is 50–100K vectors, and cost is not the constraint: 100K × 1,536 dims × 4 bytes is ~615 MB as float32, an unremarkable fit. Castillo’s measurement above puts the plateau two orders of magnitude lower, at 1,000 documents — so treat 50–100K as a starting point that is safe rather than one that is necessary, and let the curve below say where it actually flattens for your corpus.

Two things decide whether that number is right:

Test it as a curve, not a guess. Fit at 25K / 50K / 100K / 250K and score downstream retrieval at each — nDCG@10 or Recall@K, not explained_variance_ratio_. Explained variance says how much of the covariance survived; it does not say whether the neighbours survived. If 100K and 250K give essentially identical retrieval, 100K was reasonable.

Representativeness beats sheer size. A uniform random sample of a product catalog is dominated by its largest categories, and the projection it learns is slightly worse for the rare ones. Stratify across whatever actually varies — categories, languages, markets — before reaching for more vectors. Stratifying 100K matters more than going from 100K to 1M.

Source: a ChatGPT exchange (2026-08-10) reasoning about Doug Turnbull’s sampling advice in Principal Component Analysis - an embedding shrink-ray. The sample sizes are rules of thumb rather than measured results — the validation loop is the transferable part, not the numbers.

2. Incremental PCA (bounded memory, exact-ish)

sklearn.decomposition.IncrementalPCA fits batch by batch, so peak memory tracks the batch size rather than the corpus:

from sklearn.decomposition import IncrementalPCA
 
pca = IncrementalPCA(n_components=256, batch_size=10_000)
for batch in batches:          # pass 1 — fit
    pca.partial_fit(batch)
 
for batch in batches:          # pass 2 — transform
    reduced = pca.transform(batch)

Two passes over the data, bounded RAM. Use it when you genuinely want the full corpus in the fit — a heavily multi-modal catalog, or when a sample demonstrably shifts the spectrum.

3. Regular PCA / randomized SVD

PCA(svd_solver="randomized") avoids a full decomposition and is what the default svd_solver="auto" resolves to for large d with small n_components, but it still wants the input array in memory. Fine when the fitting set already fits — which, given option 1, it usually does.

What you must store and re-apply

The fitted artifact is the projection matrix and the mean vector. PCA centers before projecting, so the mean is part of the transform, not a detail of the fit. The batch requirement applies only to fitting: once the projection matrix and mean are saved (joblib.dump), transforming is a matrix multiply per vector and needs no batch at all.

The non-negotiable rule for retrieval: the same fitted transform must be applied to documents and to queries. Fitting one projection on documents and another on queries puts the two into different spaces and the similarity scores become meaningless.

Fitting on documents and applying to queries is the standard arrangement — stated explicitly in Turnbull’s write-up, and implied by Castillo’s methodology.

Refitting is an index schema change

The rule above has an operational consequence that is easy to miss: you cannot casually refit the projection and start applying the new transform to incoming queries. Old document vectors were projected by the old matrix and old mean; new query vectors would be projected by the new ones. The two no longer live in a compatible transformed space, and every similarity score computed across the boundary is silently wrong — no error is raised, recall just degrades.

So a newly fitted PCA is effectively an index schema/version change, not a tuning tweak. It carries the same obligations as one:

  • Version the artifact (projection matrix + mean) alongside the index it produced, and stamp every index with the transform version it was built under.
  • Refitting means reprojecting the whole corpus — a full reindex, not an incremental update.
  • Roll it out the way schema changes are rolled out: build the new index offline, then cut queries over atomically. Serving old documents against a new query transform, even briefly, is the failure mode.
  • Never mix vintages inside one index. Documents added after a refit must be projected by whichever transform that index was built under, until it is rebuilt.

This applies to any fitted transform in the pipeline, not only PCA — anything whose parameters are learned from data and then applied at query time inherits the same discipline: an ASH or ITQ learned rotation, an OPQ rotation, a trained product-quantization codebook.

Two things soften it at the edges:

  • Data-agnostic transforms still need versioning, for a weaker reason. RaBitQ and TurboQuant rotate before quantizing, but with a random rotation — nothing is fitted, so there is no drift and no refit. The rotation is still a stored artifact that documents and queries must share, so regenerating it with a different seed breaks the index just as thoroughly. What you avoid is the reason to ever want to change it.
  • Asymmetry is about quantization, not projection. Product quantization and ASH are called asymmetric because the query is left at full precision while documents are quantized. The query still goes through the same fitted projection. So “asymmetric” methods are not an exception to the rule above — they apply the shared transform and skip only the lossy rounding on the query side.

Training-time alternatives like Matryoshka Embeddings sidestep the whole problem: truncation is a fixed slice, not a fitted matrix, so there is nothing to drift.

Measured Cost on Real Embeddings

Doug Turnbull measured recall against brute-force ground truth after PCA-compressing MiniLM (384-dim) embeddings of the MS MARCO passage corpus, fitting PCA on 100K vectors (Principal Component Analysis - an embedding shrink-ray):

DimsCompressionEigenvalue %Recall
2001.9×88.50.879
1003.8×64.050.5714
507.7×42.040.2029

The degradation is steeply non-linear — roughly halving dimensions is survivable, but pushing to 7.7× collapses recall. Eigenvalue % (np.sum(eigenvalues[:k]) / np.sum(eigenvalues)) is the share of variance retained, and it tracks recall loosely rather than predicting it exactly.

The Flat-Spectrum Diagnostic

PCA only pays off where there is redundancy to harvest. If an embedding model already spreads information evenly across dimensions, the eigenvalue spectrum is flat and nothing compresses:

“Imagine the 1st eigenvalue is 15 and the 384th is 13 you’re not going to get much effective shrinkage with PCA.”

Inspect the explained-variance curve before committing to PCA — a steep drop after k components means good compressibility; a flat curve means pick a different technique. Compressibility also depends on the corpus, not just the model.

Measured Against MRL Truncation

Dylan Castillo compared PCA against MRL truncation across eight BEIR subsets, scoring nDCG@10 at the reduced dimension as a fraction of nDCG@10 at full dimensions (Honey, I Shrunk the Embeddings - Matryoshka vs PCA):

Dims3-small trunc.3-small PCAada-002 trunc.ada-002 PCA
51298%97%96%99%
25694%95%89%96%
12886%90%83%89%
6471%82%66%78%
3246%65%41%59%

The ada-002 column is the interesting one: that model predates MRL, and PCA still retained 78% at 64 dims. On this evidence PCA does not depend on the model having been trained for truncation. The two methods track each other down to 256 and diverge below.

The same study also found the projection cheaper to calibrate than usually assumed — see the fitting section above for those numbers.

These numbers are far gentler than Doug Turnbull’s recall table above, but they are not in contradiction — different model (1,536d/4,096d API models vs 384d MiniLM), different corpus, and nDCG retention rather than recall against brute-force ground truth. A higher-dimensional model has more redundancy for PCA to harvest, which is the reconciling variable worth checking first on any new model. Both were measured with exact search, so neither says how the projected space behaves inside an ANN index.

Limitations

  • Linear only — cannot capture non-linear manifolds in the data
  • Information loss is unavoidable (unless eigenvalues are zero)
  • Principal components are hard to interpret in original feature terms
  • Dimensionality Reduction — parent concept; also covers t-SNE, UMAP, Matryoshka
  • t-SNE — non-linear alternative for visualization
  • UMAP — non-linear alternative with parametric option
  • Matryoshka Embeddings — training-time alternative; dimension-flexible without projection
  • Vector Quantization — complementary compression approach
  • ASH — combines PCA with a learned rotation and scalar quantization under one bit budget
  • ITQ — PCA plus a rotation tuned for binarization; the older ancestor of that idea
  • HNSW — ANN index that benefits from reduced dimensionality
  • MS MARCO — the corpus the compression numbers above were measured on

Articles

Graph View

Backlinks