One post tagged with "Search relevance"

"You can't improve what you don't measure."

Peter Drucker, allegedly

I know. This quote gets wheeled out for everything from sales targets to step counts. Hear me out though. Search relevance is one of the places where I think it earns its keep.

I've watched a similar story play out on plenty of search teams over the years.

Someone makes a ranking change that looks obviously better than what was there before:

• A synonym list fixes a product category people kept complaining about.
• A recency boost pushes newer content higher.
• Some additional leaf queries improve precision for a handful of awkward cases.

The change fixes the queries someone happened to type while testing it. The before and after screenshots look good in the pull request, so it ships.

A few weeks later, someone notices search has quietly become worse somewhere else:

• The synonym list changed the meaning of queries in another category.
• The recency boost buried useful evergreen content.
• The extra leaf queries hurt recall and increased the number of zero-result searches.

Search relevance is still one of those areas where changes can be shipped with surprisingly little evidence.

Nobody would merge a performance change because the application felt faster. You'd want a benchmark, some measurements, and ideally a regression test. Ranking changes quite often get a handful of manually inspected queries and a "looks good to me".

Changing ranking isn't particularly difficult. Knowing what the change did to all the queries you didn't test is the harder part. The regressions usually get found eventually. They just get found through users, support tickets, analytics, or not at all when users get poor results and go elsewhere. And it generally isn't because the team doesn't care. The teams I've seen run into this problem have been good teams. They just haven't had a repeatable way to answer a fairly basic question:

"Was this actually a good change overall?"

Releval is my attempt to build the tooling I've wanted when answering that question, and today I'm releasing Releval 1.0.