Russ Cam

There's been a fair bit of interest since the release of Jev, from TypeSafe, a type of model they refer to as a System One model, taking the name from Thinking, Fast and Slow by Daniel Kahneman. System One models are designed to make fast, intuitive, structured decisions, which makes them well suited to a number of search-related tasks, including judging relevance.

Until this release, every AI judge in Releval has been a chat model. Releval sends it a prompt with the query and a result, and asks for a grade and a line of reasoning in return. This works well, and the reasoning is handy when you skim the judgments afterwards. The missing piece is some indication of how sure the model was. A 3 from a model that had no doubt looks exactly like a 3 from one that was much less certain.

The advice in the docs has always been to use AI judges to widen coverage, then sample their work and check it. That's still good advice; the harder part is deciding which judgments to sample.

Releval 1.2.0 adds AI judges that grade with Jev, which helps with exactly that. Rather than generating a grade and a line of reasoning, Jev is asked a question with an ordered set of answers and returns a probability for each one. For relevance judging, that means every judgment comes with a confidence and the distribution of probability across the grades, so the judge tells you where it was unsure, and that's a good place to start sampling. What's more, Jev can be much cheaper and faster than chat-based judges, allowing you to sample more often and get more coverage.

Until this release, an evaluation finishing in Releval produced a browser notification for the person who started it. This works well for short, interactive runs, but less well for scheduled or longer-running evaluations. If a run finished later, someone still needed to go back into Releval to see the result.

For many engineering teams, Slack is where day-to-day communication around changes already happens. Pull requests get shared there, deployment results are posted there, and people discuss whether something is ready to ship there. With the release of Releval 1.1.0, evaluation runs and AI judging outcomes can now be sent directly to a Slack channel. Notifications include the resulting metrics and, where available, how each metric changed compared with the previous run.

For example:

This gives the team some useful context without needing to open Releval each time.

When I wrote about Releval 1.0, I made the point that every query your users run is effectively another test case. You didn't choose it. It's running in production right now, and unless you're collecting what happened afterwards, you may never know how it went.

@releval/tracker 1.0 is out.

It's a browser library for collecting that second half of the story: the results users actually saw, the ones they clicked, and the conversions that followed.

Every event is sent using the open User Behavior Insights event format and carries the id of the query that produced it. A purchase three pages later can still point back to the search that started it.

The package is Apache-2.0 licensed, around 8.4 kB gzipped, and has a single runtime dependency.

There are ESM, CommonJS and script-tag builds, full TypeScript definitions, and a React adapter under @releval/tracker/react.

The interesting bit isn't really collecting a click, though, plenty of libraries can do that. It's joining that click back to the search.

"You can't improve what you don't measure."

Peter Drucker, allegedly

I know. This quote gets wheeled out for everything from sales targets to step counts. Hear me out though. Search relevance is one of the places where I think it earns its keep.

I've watched a similar story play out on plenty of search teams over the years.

Someone makes a ranking change that looks obviously better than what was there before:

• A synonym list fixes a product category people kept complaining about.
• A recency boost pushes newer content higher.
• Some additional leaf queries improve precision for a handful of awkward cases.

The change fixes the queries someone happened to type while testing it. The before and after screenshots look good in the pull request, so it ships.

A few weeks later, someone notices search has quietly become worse somewhere else:

• The synonym list changed the meaning of queries in another category.
• The recency boost buried useful evergreen content.
• The extra leaf queries hurt recall and increased the number of zero-result searches.

Search relevance is still one of those areas where changes can be shipped with surprisingly little evidence.

Nobody would merge a performance change because the application felt faster. You'd want a benchmark, some measurements, and ideally a regression test. Ranking changes quite often get a handful of manually inspected queries and a "looks good to me".

Changing ranking isn't particularly difficult. Knowing what the change did to all the queries you didn't test is the harder part. The regressions usually get found eventually. They just get found through users, support tickets, analytics, or not at all when users get poor results and go elsewhere. And it generally isn't because the team doesn't care. The teams I've seen run into this problem have been good teams. They just haven't had a repeatable way to answer a fairly basic question:

"Was this actually a good change overall?"

Releval is my attempt to build the tooling I've wanted when answering that question, and today I'm releasing Releval 1.0.