Awesome Search KG

JEV vs. LLM: Your Software Doesn't Want a Conversation. It Wants a Decision.

2 min read

JEV vs. LLM: Your Software Doesn’t Want a Conversation. It Wants a Decision.

Paywall

Medium member-only post. Key ideas only below; details are in the original. https://ai.plainenglish.io/jev-vs-llm-your-software-doesnt-want-a-conversation-it-wants-a-decision-3ca8f1c6ac76

Sajith K audits Jev’s launch claims instead of reacting to them — the only piece in the reaction that checks the vendor’s arithmetic.

Key ideas

  • The 238x price claim is true and the comparator is chosen. Exact against Claude Fable 5.1; against Claude Haiku 4.5, the realistic incumbent for classification, ~24x. Still a large margin — just not the headline number. See TypeSafe.
  • A Noul value is not a confidence score. 0.5 is a coin-flip, not medium urgency; confidence is a separate field derived from distribution peakedness. Conflating them breaks thresholds. See Jev.
  • Speed comes from parallel sampling, not a small model — questions are evaluated simultaneously, so forty cost about what one does.
  • The one genuinely independent test is mixed. A hands-on review reports ~25x faster per passage, and six of seven planted defects caught where an LLM caught all seven.
  • A third party re-reading the vendor’s dashboard is not a second measurement. He cites ~67.8% against a frontier model’s ~74.1% over 711 cases; those are TypeSafe’s own self-run four-workflow figures, and that column is agreement with frontier models, not accuracy. See How to Use Jev - A Practical Guide.
  • The vendor benchmark’s ceiling is agreement with frontier LLMs, not correctness — it uses their predictions as reference probabilities.
  • “Zero hallucinations” is the weakest claim — a type-valid answer can still be wrong, and the phrasing invites skipping evaluation.
  • Hard limits: ~32k tokens shared between state and questions, Choice cardinality 255, text only, no published p95/p99, no calibration-under-shift data, no SLA.
  • He withdraws his own framing. Every vendor use case puts the model beside an LLM, not in place of it — a decision layer around a generation layer (Staged Judging, RAG filtering, routing). His rule: if a human reads the output, you need an LLM.

Graph View

Backlinks