Releval 1.0 is here

"You can't improve what you don't measure."

Peter Drucker, allegedly

I know. This quote gets wheeled out for everything from sales targets to step counts. Hear me out though. Search relevance is one of the places where I think it earns its keep.

I've watched a similar story play out on plenty of search teams over the years.

Someone makes a ranking change that looks obviously better than what was there before:

• A synonym list fixes a product category people kept complaining about.
• A recency boost pushes newer content higher.
• Some additional leaf queries improve precision for a handful of awkward cases.

The change fixes the queries someone happened to type while testing it. The before and after screenshots look good in the pull request, so it ships.

A few weeks later, someone notices search has quietly become worse somewhere else:

• The synonym list changed the meaning of queries in another category.
• The recency boost buried useful evergreen content.
• The extra leaf queries hurt recall and increased the number of zero-result searches.

Search relevance is still one of those areas where changes can be shipped with surprisingly little evidence.

Nobody would merge a performance change because the application felt faster. You'd want a benchmark, some measurements, and ideally a regression test. Ranking changes quite often get a handful of manually inspected queries and a "looks good to me".

Changing ranking isn't particularly difficult. Knowing what the change did to all the queries you didn't test is the harder part. The regressions usually get found eventually. They just get found through users, support tickets, analytics, or not at all when users get poor results and go elsewhere. And it generally isn't because the team doesn't care. The teams I've seen run into this problem have been good teams. They just haven't had a repeatable way to answer a fairly basic question:

"Was this actually a good change overall?"

Releval is my attempt to build the tooling I've wanted when answering that question, and today I'm releasing Releval 1.0.

Releval is a platform for evaluating, tracking and improving search relevance.

You point it at a search system, give it a representative set of queries and some relevance judgments, and it runs those queries, records the results and calculates ranking metrics. Change something, run the same evaluation again, and compare what moved. That's more or less the idea.

Most of the platform exists to make that loop cheap and repeatable enough that it can become part of how ranking changes are developed rather than something done occasionally after the fact.

The loop

Everything starts with a search endpoint: the system you want to measure.

Releval doesn't particularly care what's behind it. Elasticsearch, OpenSearch, Solr, Vespa, a custom search API, or something else accessible over HTTP can all be evaluated.

Then you need three things:

1. A query set containing the queries you care about.
2. A query template describing how those queries should be sent to the search endpoint.
3. Relevance judgments describing what "good" looks like for each query.

Run an evaluation and Releval executes each query, stores the returned candidates and calculates metrics at the run, query and candidate level.

The metrics are the usual information retrieval ones: precision, recall, F-score, MAP, MRR, DCG, NDCG and ERR, calculated at whatever depth makes sense for your search experience. If users rarely look beyond the first ten results, measuring the first thousand probably isn't telling you much.

Judgments can be binary or graded, using scales from 0–4 or 0–9. Coarser scales are faster and easier to judge against; finer scales preserve more distinction between results. My preference is to use the coarsest scale that captures differences you actually care about.

A single evaluation run is useful, but comparison is where things become much more interesting. Make a change like:

• Change the analyser.
• Adjust a boost.
• Add a reranker.
• Replace the search engine entirely.

Then run the same evaluation again.

Each run keeps its configuration, results and metrics, so instead of asking:

Does search feel better?

you can ask:

Which metrics moved? Which queries improved? Which got worse? Which results changed?

Those are much better questions to be answering in a pull request, moving to an A/B test, or before shipping the change.

Judgments are the expensive bit

There isn't really a way around this: ranking metrics need relevance judgments. Somebody, or something, has to look at a result for a query and decide how relevant it is. Those judgments are some of the most valuable data you can have about a search system, but producing them can also be tedious and expensive. Releval helps to reduce that cost without pretending it doesn't exist.

You can judge results manually using a keyboard-driven interface designed for working through a judgment backlog quickly. You can import grades you already have as a judgment list, whether they came from click data, a previous annotation exercise, another evaluation system, or a spreadsheet somebody has been maintaining. You can also use an LLM-as-a-judge for results that haven't been graded yet. Releval supports OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Ollama and OpenAI-compatible endpoints.

These sources are intended to work together rather than being mutually exclusive. For the same query and result, Releval prefers a deliberate human judgment over an AI-generated judgment. Existing data can provide coverage without allowing a lower-quality source to silently replace a better one.

AI judging is useful here, but I don't think it's useful to pretend that it solves relevance judgment entirely. Its main advantage is coverage. If you have two thousand queries, having a model make a first pass can be much more practical than manually judging every result before you can calculate anything. You can then spend human attention on the queries that are important, ambiguous or where the model appears to get things wrong. That's generally the way I'd use it.

Judgments also carry forward between runs. Once a candidate has been judged for a query, Releval can reuse that judgment rather than asking you to grade the same result every time you adjust a boost.

The headline number isn't enough

Suppose overall NDCG goes up. That's useful to know, but it isn't enough to tell you whether the change was good. Maybe most queries improved slightly while several important ones regressed badly. Or perhaps a large number of low-value queries improved and masked a regression in the small group of queries that matter most to the business.

Two implementations can have very similar aggregate scores while behaving quite differently query by query. Metrics tell you where to look. You still need to understand what changed. Releval provides a few different ways to do that.

Query Lab is intended for the fast development loop. Run a query, change the template, inspect the results, add judgments and compare against a baseline. Once you're happy with the behaviour for the queries you're working on, run the full evaluation.

I think of Query Lab as the workbench and an evaluation run as the regression test.

The dashboard tracks metrics against previous runs and over time. It also records where the evaluation configuration changed. This is useful because if the metrics suddenly move, you want to know whether the search implementation changed or whether somebody changed the query set, judgments or evaluation configuration.

You can also compare two runs head to head: aggregate metrics, per-query deltas and the individual candidates that moved. That makes it possible to go from "the average improved" to understanding where that improvement came from, what regressed, and whether the ranking changed in the way you intended.

Sometimes the right result is "nothing changed"

Not every search change is intended to affect relevance. You might be migrating from Solr to Elasticsearch, replacing an API, introducing a cache, upgrading a search engine, or changing infrastructure. In those cases, the goal may be that users see exactly the same results.

Releval can compare ranked result lists directly using similarity metrics such as Rank-Biased Overlap, without requiring relevance judgments. That lets you distinguish between changing the implementation and changing what users actually see. During a migration, that's a useful distinction to be able to measure rather than assume.

Put it in the pipeline

Everything available through the UI is also available through APIs. Releval has been built from the beginning with automation in mind.

App clients provide machine-to-machine access for CI and other automation, and there's an MCP server for using Releval through tools and AI assistants.

The important bit for me is making relevance evaluation easy enough to run as part of normal development. Software testing became ubiquitous partly because we made it cheap and routine. You change some code, tests run, and you find out whether behaviour you care about changed. I'd like ranking changes to work much the same way.

A pull request changes ranking behaviour. An evaluation runs against an accepted baseline. You can see which metrics moved and which queries regressed before the change reaches production. It doesn't need to be a special relevance exercise or a weekly review. It can just be another check associated with changing the system.

You're already testing in production

Every query your users run is effectively another test case. The difference is that you probably didn't choose it, and you may not be looking at the result.

Releval has integrated User Behavior Insights for capturing queries and subsequent behaviour using the open UBI standard: clicks, abandonment, reformulations and other search interactions. This data is useful in two ways:

• As a source of implicit relevance information.

and arguably more useful,

• As a means to discovering which queries actually deserve to be in your evaluation set.

The queries your users care about are rarely identical to the examples somebody came up with while building the search system. User Behavior Insights therefore gives you a feedback loop:

Observe what people search for. Find important or poorly performing queries. Add them to an evaluation. Change the ranking. Measure the result. Keep watching.

Over time, the evaluation set becomes less of a static test fixture and more of a record of the search problems you care about.

Running it

Releval runs in your own infrastructure, so queries, results and judgments don't need to leave your environment.

The quickest way to get started is Docker Compose. The Getting started guide covers installation and running a first evaluation.

If you'd rather watch than read, the video below goes through the complete flow: connecting a search system, creating a query set, evaluating two query templates, generating judgments with an AI judge, and comparing the runs.

That's 1.0

Releval is the platform I wanted when I've had to work out whether a ranking change actually made search better. Ranking changes deserve the same kind of evidence we'd expect when changing other important application behaviour. Releval is my attempt to make collecting that evidence straightforward enough that it can happen on every change.

The introduction covers the main concepts, Getting started will take you through a first evaluation, and the community repository is the place to report problems, suggest improvements and discuss where the platform should go next.

I'd particularly like feedback from people building and operating real search systems. If something doesn't fit the way you work, I'd like to hear about it. Those gaps are generally more useful than hearing that everything worked as expected.

Give Releval a try, and let me know how you get on. 😎