Late Interaction in OpenSearch
How Late Interaction models — ColBERT for text, ColPali for document images — are indexed, scored, and scaled inside OpenSearch. Native support landed in OpenSearch 3.3 (announced December 2025) via the lateInteractionScore scoring function, with a parallel Lucene-level path (LateInteractionField, Lucene 10.3+) feeding it. As in Elasticsearch, late interaction is positioned as a reranker over a fast single-vector first stage — see the parallel topic Late Interaction in Elasticsearch.
The Building Blocks
Multi-vector storage: object + float
Unlike Elastic’s dedicated rank_vectors field, OpenSearch stores token-level multi-vectors using a composite of the object and float field types — “storage and retrieval of token-level vector embeddings used in late interaction models.” Token vectors must share a consistent dimensionality, but documents may carry varying numbers of vectors.
lateInteractionScore (painless)
Scoring is a painless script function with the same MaxSim semantics as Elastic’s maxSimDotProduct: for each query vector, take the max similarity over all document vectors, then sum.
lateInteractionScore(params.query_vectors, 'my_vector', params._source, params.space_type)
Lucene path (LateInteractionField, 10.3+)
Since Lucene 10.3, LateInteractionField accepts float[][] multi-vector embeddings, encodes them as binary, and indexes them as BinaryDocValues. LateInteractionRescorer implements the default sum(max(vectorSimilarity)). An open k-NN proposal (issue #2934) tracks deeper Lucene-based rescoring.
The Pipeline
OpenSearch wires late interaction through two ML-inference processors:
| Processor | Role |
|---|---|
| ml-inference ingest processor | At ingest, generates both the single-vector (bi-encoder) embedding and the multi-vector late-interaction embedding |
| ml-inference search request processor | Rewrites the incoming query into a k-NN query plus a lateInteractionScore rescore |
Two-Phase Retrieval
- Phase 1 — fast approximate k-NN over single-vector bi-encoder embeddings selects a small candidate set.
- Phase 2 — rerank those candidates with
lateInteractionScoreover the token-level multi-vectors.
The Cost
Storage requirements increase 10–100× versus single-vector retrieval — the same fundamental pressure late interaction creates everywhere. OpenSearch’s mitigation story is younger than Elastic’s: there is no built-in rank_vectors-style bit/average-vector toolkit yet; compression leans on the staged retrieve-then-rerank pattern and the underlying k-NN plugin (Faiss / nmslib / Lucene HNSW) for the cheap first stage.
MUVERA as the first stage
An AWS sample (Scaling Late Interaction Retrieval in OpenSearch with MUVERA) replaces the bi-encoder first stage with a MUVERA fixed dimensional encoding of the multi-vectors themselves, indexed as a knn_vector and reranked with lateInteractionScore. On self-managed OpenSearch 3.8.0 the opensearch-muvera-plugin computes it server-side (muvera / muvera_query processors); on Amazon OpenSearch Service it is computed client-side with FastEmbed. FDE dimension is capped at 16,000.
Its benchmarks: MUVERA + rerank keeps 90–97% of exact MaxSim nDCG@10 on text (SciFact, NFCorpus) and 87% of recall@10 on visual pages (IRPAPERS), against 28–52% for mean-pooling + rerank. Rerank is ~97% of query latency; a richer FDE (r_reps 30) lets the rerank depth halve at equal quality for ~40% less latency. int8 multi-vectors cut storage 4× at near-zero quality cost but no speedup while scoring reads from _source; a native LateInteractionField kernel was ~34× faster in a Lucene microbenchmark (k-NN RFC #3439).
OpenSearch vs Elasticsearch for Late Interaction
| OpenSearch (3.3+) | Elasticsearch (8.18+) | |
|---|---|---|
| Multi-vector field | object + float composite | rank_vectors (dedicated) |
| MaxSim scoring | lateInteractionScore painless fn | maxSimDotProduct / maxSimInvHamming |
| Lucene primitive | LateInteractionField / LateInteractionRescorer | rank_vectors codec |
| First-stage approx | single-vector k-NN | average vectors (+ BBQ) |
| Built-in compression | (none yet) | bit vectors, average vectors, Token Pooling |
| Two-stage mechanism | ml-inference search processor | rescore retriever |
| Role | reranker | reranker |
Both converge on the same architecture; Elastic is further along on native multi-vector compression, OpenSearch has just reached parity on the core scoring primitive. See Elasticsearch vs OpenSearch for the broader engine comparison.
Where Late Interaction Fits
| Bi-Encoder | Late Interaction (ColBERT/ColPali) | Cross-Encoder | |
|---|---|---|---|
| Vectors compared | 1 vs 1 | many vs many (MaxSim) | full transformer pass |
| Storage | Low | High (10–100×) | None (stateless) |
| Latency | Fast | Medium | Slow |
| Role in OpenSearch | first-stage k-NN | rerank top-k | rerank top-k |
Related Concepts
- Late Interaction — the architecture (MaxSim over per-token/per-patch vectors)
- ColBERT — text-domain late interaction
- ColPali — visual late interaction (document page images)
- HNSW — k-NN plugin index for first-stage candidates
- Reranking — late interaction’s role in OpenSearch
- Bi-Encoder · Cross-Encoder — the no/early-interaction endpoints
Related Topics
- Late Interaction in Elasticsearch — the parallel Elastic implementation
- Late Interaction in Qdrant — the vector-DB take (MUVERA prefetch + ColBERT rerank)
- Late Interaction in Vespa — tensor-expression MaxSim and billion-scale ColPali
- Interaction Paradigms — the no/late/early interaction spectrum
- Elasticsearch vs OpenSearch · Search Platforms — the engine landscape
- Reasoning Reranking · Frontier of Search 2025
Source
-
Boost search relevance with late interaction models — OpenSearch blog (Dec 2025; Mingshi Liu, Vigya Sharma, Navneet Verma, Brian Flores)
-
k-NN issue #2934 — Lucene-based rescoring with late interaction
-
Scaling Late Interaction Retrieval in OpenSearch with MUVERA — aws-samples MUVERA plugin, client-side FDE path, and OpenSearchCon benchmarks