Awesome Search KG

Late Interaction

4 min read

Late Interaction

Definition

Late interaction is a retrieval architecture where query and document representations are computed independently (like Bi-Encoder) but interact at the token level during scoring (unlike single-vector bi-encoders). The best-known implementation is ColBERT.

How It Works

Query  → Encoder → [q₁, q₂, ..., qₘ]  (per-token vectors)
Doc    → Encoder → [d₁, d₂, ..., dₙ]  (per-token vectors)

Score = Σᵢ max_j (qᵢ · dⱼ)    (MaxSim)

The MaxSim operator finds, for each query token, the document token most similar to it, then sums these maximum similarities.

The max_j is itself a pooling step — but over the similarity matrix, not over representations. It collapses the m × n score matrix to m along the document-token axis, which is then summed to a scalar. Worth naming explicitly, because it is easily confused with the max pooling that a Bi-Encoder may use to build its single vector: that one pools before comparison and discards information permanently at index time, whereas MaxSim pools after comparison, at query time, with the full token vectors still in hand. Deferring that collapse is the whole point of late interaction.

Interaction Timeline Comparison

Late interaction is one point on a broader axis — when the query and document interact. See Interaction Paradigms for the full no/late/early spectrum.

Early interaction  (Cross-Encoder):  [Q + D] → Encoder → score
No interaction     (Bi-Encoder):     Q → enc → q; D → enc → d; score(q,d)
Late interaction   (ColBERT):        Q → enc → [q_tokens]; D → enc → [d_tokens]; MaxSim

Key Properties

  • Documents can be pre-encoded (only token-level vectors stored per doc)
  • Token-level scoring captures fine-grained relevance
  • Requires more storage than single-vector bi-encoders (|tokens| × dim per document)
  • Quality approaches Cross-Encoder while remaining scalable

Compression

Since late interaction stores many vectors per document, storage is a concern:

  • ColBERT uses 128-dim token vectors
  • Vespa’s implementation uses int8 compression (32x reduction)
  • Residual compression in ColBERTv2 reduces storage 6-10x
  • Token Pooling — clusters similar token vectors and replaces each cluster with its mean; pool factor 3 retains 97.8% performance while cutting vectors by 66.7%
  • Bit vectors — sign quantization (>0 → 1); 32× storage reduction; pairs with hamming distance (maxSimInvHamming)
  • Average vectors — single normalized mean vector per document; HNSW-indexable but loses per-token granularity

Multimodal Extension: ColPali

ColPali applies late interaction to document page images (PDFs, slides) using a PaliGemma backbone. Generates ~1000 patch-level vectors per page instead of token-level vectors. Same MaxSim scoring, much higher storage pressure. Used primarily as a reranker over HNSW-retrieved candidates.

  • ColBERT — primary text-domain implementation of late interaction
  • ColPali — visual-domain late interaction (document page images)
  • Token Pooling — compression for multi-vector late interaction embeddings
  • Pooling — the single-vector collapse late interaction exists to defer; also home of the disambiguation, in which MaxSim is the score-level sense
  • MUVERA — fixed-dimensional single-vector approximation enabling ANN first-stage retrieval
  • Bi-Encoder — no interaction model (the baseline)
  • Cross-Encoder — early interaction (joint encoding)
  • Dense Vector Retrieval — late interaction uses multi-vector dense representations

Articles

Graph View

Backlinks