Evaluating Search: Measuring Searcher Behavior
Source: https://medium.com/@dtunkelang/evaluating-search-measuring-searcher-behavior-5f8347619eb0 Author: Daniel Tunkelang Series: Evaluating Search (Part 2 of 4)
Summary
Frames search evaluation as measuring searcher ROI: the return (what they find) divided by the investment (effort they expend). Behavioral signals provide implicit, scalable feedback on search quality without needing human judgments.
The Searcher ROI Framework
Return indicators (what they find)
- Conversions — purchase, signup, etc. Strong signal but sparse
- Clicks — weaker but abundant. “Clicks tend to overestimate return, while conversions underestimate it” — use both together
Investment indicators (effort expended)
- MRR — position of first clicked result; lower rank = more user effort
- Search frequency — how often users must search to complete tasks
- Typing volume — characters typed; autocomplete reduces investment
Key Insight
Multiple metrics paint a fuller picture than any single metric. Discrepancies between metrics (e.g., high CTR but low conversion) reveal counterintuitive, actionable patterns.
“All models are wrong, but some are useful.”
Series
- Evaluating Good Search - Measure It
- Measuring Searcher Behavior (this article)
- Evaluating Search - Using Human Judgments
- When There’s No Conversion Rate