Dataset Types for Embedding Fine-Tuning
Embedding fine-tuning data comes in a small number of shapes, and the shape decides which loss can be used. Ritik Jain groups them into six. In practice the choice is rarely free: it is set by what labels already exist — query logs, human similarity scores, relevance judgments — and the loss follows from it (see Embedding Loss Functions).
The six shapes
| Type | Record | Typical use | Loss | Suggested minimum |
|---|---|---|---|---|
| Positive pairs | (anchor, positive) | semantic search, document retrieval, FAQ | MultipleNegativesRankingLoss | 2,000–5,000 pairs |
| Triplets | (anchor, positive, negative) | metric learning, ranking with explicit negatives | TripletLoss or MNRL | 1,000–5,000 triplets |
| Similarity-score pairs (STS) | (sentence 1, sentence 2, score 0–5) | sentence similarity, paraphrase | CoSENTLoss | 2,000–5,000 pairs |
| Natural-language inference | (premise, hypothesis, entailment / neutral / contradiction) | logical relationships | MNRL, after conversion to triplets | 10,000+ |
| IR ranking | (query, document, relevance grade), TREC qrels | document ranking, passage retrieval, QA | MNRL | 100+ queries over 10k–100k documents |
| Hard negatives | (query, positive, several hard negatives) | robustness, edge cases | MNRL | derived from the others |
The minimums are the author’s guidance, not measured thresholds.
What makes each one usable
- Pairs — the positive must actually be relevant; no exact duplicates; anchor and positive lengths kept in a sensible range; one domain throughout.
- Triplets — the negative should be clearly wrong yet ideally hard; no overlap between positive and negative; two to five negatives per anchor.
- STS pairs — scores from human annotators rather than generated, with inter-annotator agreement of at least 0.8 and a spread of scores across the scale rather than clusters at the ends.
- NLI — the three labels in roughly equal proportion, agreement above 0.7, no ambiguous examples.
- IR ranking — queries that reflect real user intent, judgments with agreement above 0.7, a balance of relevant and non-relevant documents, and no query shared between train and test. That last rule is the one that most often inflates results — see Model Selection and Fine-Tuning Evaluation.
- Hard negatives — genuinely difficult but never secretly correct, kept separate between train and test, and cheap enough to generate at scale.
Where hard negatives come from
The author lists five strategies: top-k results from BM25 or the current model, statements that contradict the answer, partially correct but incomplete answers, same-topic text from the wrong domain, and a curriculum that starts easy and gets harder. The false-negative risk — a “negative” that is actually relevant — is covered in Hard Negative Mining.
How these relate to other sources of labels
IR ranking data is the same material as Judgment Lists. Positive pairs are often mined from behaviour — clicks or purchases treated as positives, see Implicit Judgments — or generated by an LLM to bootstrap (Embedding Fine-tuning compares the construction methods). What is chosen as a positive defines what “relevant” means to the resulting model (Contrastive Learning).
Related Concepts
- Embedding Loss Functions — which loss each shape pairs with
- Hard Negative Mining · Contrastive Learning
- Embedding Fine-tuning · Domain-Specific Semantic Gaps
- Judgment Lists · Implicit Judgments
- Hyperparameter Tuning for Embedding Fine-Tuning
Related Topics
- Model Selection and Fine-Tuning Evaluation — split discipline and leakage