2026.38

History — 2026 week 38 (Sep 14 – Sep 20, 2026)

Newest first.

2026-09-20 — How good are Jev’s probabilities, actually (7 new, 8 updated)

A model that answers in probabilities invites one obvious question nobody had answered: are the probabilities any good? Adapting Jev to Your Domain with GEPA measures them, and at Jev’s default prompt they are not — a 10-bin ECE of 0.173 against 0.052 for a threshold-tuned TF-IDF baseline over 300 sentences of ADE Corpus V2, with the confidence field returning exactly 1.0 on half the test set and ten of those answers wrong. Rewriting the instruction with GEPA, a reflective prompt optimizer pointed at Brier score, moved F1 from 69.1% to 79.7% on a fresh split of the same corpus. Calibration turns out to be partly a property of the prompt — from Praneeth Paikray, with executed notebooks.

Corrections

• Reception of Jev recorded that no expected-calibration-error figure or reliability evidence existed for Jev beyond one retrieval benchmark. One now does, from outside the vendor, and it goes against the model; that paragraph has been rewritten.

New — Articles — Adapting Jev to Your Domain with GEPA New — Concepts — Brier Score · Expected Calibration Error · Prompt Optimization New — People — Praneeth Paikray New — Tools — GEPA (reflective prompt optimizer) New — Datasets — ADE Corpus V2 (adverse-drug-event sentence classification) Updated — Jev (domain adaptation; the calibration and latency readings) · TypeSafe (latency and calibration claims measured from outside) · Reception of Jev · Calibrated Relevance Probability (calibration is partly a property of the prompt) · Prompt Sensitivity · Staged Judging (measuring the bands, not the metric) · DSPy · Hev meets Jev

2026-09-20 — What practitioners actually said about Jev (3 new, 11 updated)

Named practitioners split over Jev along a line that is not approval: almost nobody disputes the speed or the price. Daniel Tunkelang ran it for a day and reported frontier-model quality at less than 1% of the cost. Doug Turnbull reads it as what finally makes pairwise judgment cheap enough to prefer over a 3/5-or-4/5 scale. Andreas Wagner grants the efficiency and denies the rest — a confident wrong answer is still the problem. Sajith K checked the arithmetic: the 238x price claim is exactly right against a frontier model nobody uses for classification, and roughly 24x against the one they do.

New — Articles — JEV vs LLM - Your Software Doesn’t Want a Conversation It Wants a Decision New — Topics — Reception of Jev New — People — Sajith K Updated — Cross-Encoder (the score is ordinal by training, not by architecture) · Jev (a Noul value is not a confidence score) · TypeSafe (the price claim under audit) · Pairwise Relevance Evaluation · Daniel Tunkelang · Doug Turnbull · Andreas Wagner · Hev · Sai Yashwanth · Prosper Otemuyiwa · Annabell Schäfer

Corrections

• Cross-Encoder stated flatly that a cross-encoder’s score is not a probability. The architecture permits a sigmoid head fit with binary cross-entropy; what leaves the score ordinal in practice is the ranking objective and the hard-negative sampling behind it, and the note now says so.
• The entry titled A decision model keeps up with the rerankers sold a threshold portable across corpora as a property in hand. It rests on one BEIR subset, and the entry now says so — it also no longer claims parity with the purpose-built rerankers, and carries the 40-query sample size behind TypeSafe Cookbook - Re-ranking’s 5%-to-18% top-1 move.

2026-09-19 — Sampling a judged set is an artifact of what judging cost (3 new, 3 updated)

Judged sets get sampled because grading every case with a generative model is expensive, and Jevals argues the constraint has moved. Grade the whole set with a calibrated decision model, set a bar on the confidence each verdict carries, and escalate only the unclear band — full coverage instead of a subset extrapolated to the whole. It also names why a generative judge’s confidence misleads: the model picks a token standing for its verdict, then writes a justification for the verdict it already picked, so the rationale explains the decision rather than producing it. From Jev - The Most Interesting Model Released This Year, an opinion piece with no measurements; the framework calls itself a research preview.

New — Jev - The Most Interesting Model Released This Year · Jevals · Sai Yashwanth Updated — LLM as Judge (the rationalisation problem; grading everything instead of sampling) · Staged Judging · Jev

Corrections

• The Judging Without Generation section added to LLM as Judge earlier this week was nested under The Economics Problem rather than standing on its own; it now sits at the top level where it was meant to.

2026-09-19 — Three practical accounts of judging by probability (8 new, 8 updated)

Reranking and eval scoring are the same job, and three write-ups now say so from different angles. TypeSafe Cookbook - Re-ranking judges one candidate at a time over CLERC legal citations, where a BM25 shortlist held the correct passage for every one of 40 queries and ranking alone moved top-1 from 5% to 18% — retrieval flawless, ordering the whole bottleneck, the inverse of the usual complaint. Using TypeSafe’s Jev for Evals points the same primitive at rubric verdicts: a vague criterion returns low confidence rather than a confident label. How to Use Jev - A Practical Guide collects the documented failure modes and states the sharpest limit — a calibrated score describes the verdict, never the evidence behind it.

New — TypeSafe Cookbook - Re-ranking · Using TypeSafe’s Jev for Evals · How to Use Jev - A Practical Guide · CLERC · Langfuse · Valyu AI · Annabell Schäfer · Prosper Otemuyiwa Updated — Reranking (perfect recall with ranking still failing) · LLM as Judge (judging without generation; abstention, rationale, agreement-is-not-accuracy) · Calibrated Relevance Probability (calibration is not grounding) · Jev · TypeSafe · Staged Judging · Diogo Almeida · Hev meets Jev

Corrections

• Jev’s request budget was recorded flatly as 32k. It is documented inconsistently — the models page gives 64k per request and 32k for state plus the longest question, while OpenRouter lists 32k — and the note now says so rather than picking one.
• Diogo Almeida is described in third-party coverage as co-inventor of RLHF and InstructGPT. His own account is narrower, and the note follows his wording.

2026-09-19 — Where the calibrated probability comes from (4 new, 5 updated)

Calibration is a training objective, not a by-product — which is why this model’s score is a probability where a ranking-trained Cross-Encoder’s logit is not. System One Models fix their output space before the call and sample every answer in parallel, and Reinforcement Learning for Calibrated Decisions trains for confidence that tracks accuracy rather than for human preference or a verifier. Diogo Almeida, previously at OpenAI, founded the lab on the premise that chat-shaped models are the wrong interface for software. Read as a vendor announcement: the architecture claims are checkable, the parity claims rest on a benchmark its author designed and scored, and Introducing System One Models & Jev records both with the post’s own caveats.

New — Introducing System One Models & Jev · System One Model · Reinforcement Learning for Calibrated Decisions · Diogo Almeida Updated — Jev (training, pricing, cardinality and request-budget limits) · TypeSafe (founding, and how to read its benchmark) · Calibrated Relevance Probability (where calibration comes from) · Hev meets Jev · Reasoning Reranking

Corrections

• Jev and TypeSafe were written from secondary sources and documentation; both now rest on the primary announcement. The latency envelope quoted for Jev is the vendor’s 70–500 ms for its own workloads — not what the thirty-document rerank call in Hev meets Jev costs, which reached a p95 near 1.4 s.

2026-09-19 — A decision model keeps up with the rerankers (7 new, 6 updated)

A model built to emit typed decisions rather than text reranks about as well as the models built to do nothing else. Jev, TypeSafe’s structured-output model, answers one true-or-false question per candidate and returns a probability; over a BM25 top-30 shortlist it reached 0.501 mean nDCG@10 across three BEIR subsets, against 0.504 for Voyage rerank-3 and 0.404 unreranked — untuned, on a generic prompt. The more durable point is Calibrated Relevance Probability: a score to prune and fuse with, which a Cross-Encoder logit cannot do and nDCG cannot detect — though a threshold that survives a change of corpus is so far a promise, measured on one subset. General LLMs matched the quality listwise but at seconds per query.

New — Hev meets Jev · Calibrated Relevance Probability · Jev · hev-rerank · TypeSafe · Mixedbread · Hev Updated — Reranking (structured-decision reranker type) · Cross-Encoder (logit vs probability) · Reasoning Reranking · BEIR · Voyage AI · Cohere

Corrections

• Reranking carried an empty Related Concepts heading with its link list stranded at the end of the preceding section; the list has been moved back under its own heading.

2026-09-16 — Expected Reciprocal Rank and picking a metric (1 new, 5 updated)

NDCG credits every relevant document in a list independently; ERR assumes the user reads top-down and stops once satisfied, so a strong result sitting under another strong result earns almost nothing. ERR now covers that cascade model — the formula, where its discount comes from, and why navigational and known-item search want it while e-commerce and exploratory queries want NDCG. It sits between MRR, which it generalizes to graded relevance, and NDCG, which drops the stopping model entirely. Drawn from Demystifying nDCG and ERR; Relevance Program Setup now points at that post and at Choosing Your Search Relevance Evaluation Metric for the selection step it describes.

New — ERR (Expected Reciprocal Rank) Updated — Relevance Program Setup (metric-selection sources linked) · NDCG · MRR · Demystifying nDCG and ERR · Choosing Your Search Relevance Evaluation Metric