2 posts tagged with "Search relevance"

There's been a fair bit of interest since the release of Jev, from TypeSafe, a type of model they refer to as a System One model, taking the name from Thinking, Fast and Slow by Daniel Kahneman. System One models are designed to make fast, intuitive, structured decisions, which makes them well suited to a number of search-related tasks, including judging relevance.

Until this release, every AI judge in Releval has been a chat model. Releval sends it a prompt with the query and a result, and asks for a grade and a line of reasoning in return. This works well, and the reasoning is handy when you skim the judgments afterwards. The missing piece is some indication of how sure the model was. A 3 from a model that had no doubt looks exactly like a 3 from one that was much less certain.

The advice in the docs has always been to use AI judges to widen coverage, then sample their work and check it. That's still good advice; the harder part is deciding which judgments to sample.

Releval 1.2.0 adds AI judges that grade with Jev, which helps with exactly that. Rather than generating a grade and a line of reasoning, Jev is asked a question with an ordered set of answers and returns a probability for each one. For relevance judging, that means every judgment comes with a confidence and the distribution of probability across the grades, so the judge tells you where it was unsure, and that's a good place to start sampling. What's more, Jev can be much cheaper and faster than chat-based judges, allowing you to sample more often and get more coverage.

"You can't improve what you don't measure."

Peter Drucker, allegedly

I know. This quote gets wheeled out for everything from sales targets to step counts. Hear me out though. Search relevance is one of the places where I think it earns its keep.

I've watched a similar story play out on plenty of search teams over the years.

Someone makes a ranking change that looks obviously better than what was there before:

• A synonym list fixes a product category people kept complaining about.
• A recency boost pushes newer content higher.
• Some additional leaf queries improve precision for a handful of awkward cases.

The change fixes the queries someone happened to type while testing it. The before and after screenshots look good in the pull request, so it ships.

A few weeks later, someone notices search has quietly become worse somewhere else:

• The synonym list changed the meaning of queries in another category.
• The recency boost buried useful evergreen content.
• The extra leaf queries hurt recall and increased the number of zero-result searches.

Search relevance is still one of those areas where changes can be shipped with surprisingly little evidence.

Nobody would merge a performance change because the application felt faster. You'd want a benchmark, some measurements, and ideally a regression test. Ranking changes quite often get a handful of manually inspected queries and a "looks good to me".

Changing ranking isn't particularly difficult. Knowing what the change did to all the queries you didn't test is the harder part. The regressions usually get found eventually. They just get found through users, support tickets, analytics, or not at all when users get poor results and go elsewhere. And it generally isn't because the team doesn't care. The teams I've seen run into this problem have been good teams. They just haven't had a repeatable way to answer a fairly basic question:

"Was this actually a good change overall?"

Releval is my attempt to build the tooling I've wanted when answering that question, and today I'm releasing Releval 1.0.