@releval/tracker 1.0 is out.

It's a browser library for collecting that second half of the story: the results users actually saw, the ones they clicked, and the conversions that followed.

Every event is sent using the open User Behavior Insights event format and carries the id of the query that produced it. A purchase three pages later can still point back to the search that started it.

The package is Apache-2.0 licensed, around 8.4 kB gzipped, and has a single runtime dependency.

There are ESM, CommonJS and script-tag builds, full TypeScript definitions, and a React adapter under @releval/tracker/react.

The interesting bit isn't really collecting a click, though, plenty of libraries can do that. It's joining that click back to the search.

Four events, not analytics​

@releval/tracker isn't intended to be a general-purpose browser analytics library. For search, there are four things we're interested in collecting:

• a search, when someone runs a query;
• an impression, when a result is actually seen;
• a click, when someone chooses one; and
• whatever conversion follows, which might be a view, an add to cart, a purchase, or something else relevant to your domain.

Events use the open UBI event format. A click looks like this on the wire:

{
  "action_name": "click",
  "timestamp": "2026-09-07T10:14:22.113Z",
  "application": "relevaltech-html",
  "site_id": "01K4S...",
  "query_id": "01K4S...",
  "session_id": "01K4S...",
  "client_id": "01K4S...",
  "event_attributes": {
    "position": { "ordinal": 3 },
    "object": { "object_id": "NT-001", "object_id_field": "product_id" }
  }
}

You tell the tracker which interactions you're interested in and how to find the relevant data on the page. It takes care of turning those interactions into events and sending them to Releval. With the collection part covered, we need to be able to stitch this all together.

It all hangs on query_id​

A click on its own tells you somebody clicked something. That's useful, but not particularly useful for understanding search. A click carrying the query_id of the search that produced it, the object that was clicked, and the ordinal where it appeared is something you can actually compute with.

Now you can ask things like:

• Which queries get results but no clicks?
• How does click-through rate change by rank?
• Which searches eventually lead to a purchase?
• Which queries are people repeatedly reformulating?

The common thread is that you need to know which search produced the interaction. That is what query_id is for. The catch is that the browser can't mint a query_id on its own.

Your application is already sitting between the user and the search engine, so it's also the one place that knows both sides of the search: what was asked and what came back.

It registers that pair with Releval and passes the ids down with the results it was returning anyway. The query_id can come from your system or Releval:

• If you already have a request id, trace id or some other identifier worth correlating against, supply it and Releval will use it.
• If you don't, leave it out and Releval generates one.

Either way, the important thing is that the same value makes it back to the page and onto everything that follows.

Who sends what​

Once the page has the query_id, the rest looks like this:

There are a couple of details to focus in on.

Impressions and clicks come from the browser because the browser is the thing that knows whether a result was actually visible and what the user clicked. Trying to infer an impression from "the server returned this result" is not the same thing. Result 37 may have been returned. That doesn't mean the user saw it.

The checkout at the bottom is different. By that point, the server knows what was actually bought, so the conversion can be sent from the backend system. Both events from the frontend and backend land against the same query_id.

That's the whole model in a nutshell: collect events where the truth about those events lives, and carry enough context through to join them afterwards.

Get the query_id from your backend to your page and most of this becomes fairly mechanical. Miss it and you have a pile of interactions you can count but can't attribute.

Four steps and one check​

The @releval/tracker repository ships two example storefronts. The walkthrough below takes the plain HTML one through the full flow against a real Releval deployment, from registering a Site through to querying the events that arrived:

Doing the same thing on your own site is four steps. And one check that is very easy to forget.

1. Register a Site​

First, register a Site in Releval under Insights and give it the origins your search pages are served from:

https://shop.example.com
https://staging.shop.example.com

That gives you a site_id. The site_id is public and is intended to live in the page. It isn't a secret. The origin allow list mitigates some unrelated website from pointing its tracker at your Releval deployment and spamming it with events. If an event comes from an origin that isn't on the list, it is dropped.

2. Start the tracker​

Install the tracker and point it at Releval:

import { Tracker } from '@releval/tracker';

export const tracker = new Tracker({
  application: 'web-search',
  endpointHost: '${RELEVAL_HOST}',
  siteId: 'YOUR_SITE_ID',
});

tracker.start();

application is worth thinking about rather than typing web and forgetting about it. It's what lets you distinguish your website search from your mobile app search, or one experiment from another, when you come back to analyse the events later.

3. Register the query​

Wherever the search happens on your backend, register the query with Releval and return the query_id with the search results. This looks something like:

const results = await mySearchEngine.search(userQuery);

const { query_id } = await releval.trackQuery({
  application: 'web-search',
  user_query: userQuery,
  client_id: clientIdFromTheBrowser,   // tracker.clientId, sent with the search request
  // query_id: requestId,              // optional: your own id, or Releval generates one
});

return { results, query_id };

One easy thing to leave out here is tracker.clientId. The browser creates it, so it needs to go up with the search request and then into trackQuery. Without it, the query and the later events can still be joined through query_id, but Releval can't tell that separate searches came from the same browser.

4. Tell the tracker what a result is​

For a server-rendered page, this can be little more than some data attributes:

<ul data-query-id="01K4S..." data-query="wireless headphones">
  <li data-object-id="SKU-1" data-ordinal="1">...</li>
  <li data-object-id="SKU-2" data-ordinal="2">...</li>
</ul>

Then wire up clicks and impressions:

tracker.trackResultClicks({
  selector: '[data-query-id] [data-object-id]',
  ignore: '[data-add-to-cart]',
});

tracker.trackResultImpressions({
  selector: '[data-query-id] [data-object-id]',
});

That's the whole result-tracking setup for a simple server-rendered page. The collectors watch clicks and viewport impressions and turn them into UBI events using the attributes already on the result.

The ignore is worth calling out. Say your result card contains an add-to-cart button. Without that exclusion, clicking the button may count as both an add-to-cart and a result click. Congratulations, your CTR just improved!

For React applications, the @releval/tracker/react package takes a slightly more React-shaped approach with a SearchResults context and an impression hook.

5. Check that anything arrived​

This one is worth making a step in its own right because the ingest path is deliberately quiet. track-event responds with 202 whether the event is accepted or not.

An unknown site_id, an origin that isn't allow-listed and a malformed event can therefore all look like success from the browser. The reason is logged server-side instead. That behaviour is deliberate because the analytics endpoint shouldn't become a side channel for probing the deployment, and collecting an event shouldn't interfere with the application somebody is actually trying to use.

It does mean that staring at a green request in DevTools doesn't tell you very much. The Site activity column in Releval tells you whether events are actually arriving.

Then you can ask questions​

Once there is some data, the Insights query workspace lets you query the underlying UBI events directly. For example, click-through rate by rank:

SELECT toInt32(event_attributes.position.ordinal) AS rank,
       countIf(action_name = 'impression') AS impressions,
       countIf(action_name = 'click') AS clicks,
       round(clicks / impressions, 3) AS ctr
FROM ubi_events
WHERE rank > 0
GROUP BY rank
ORDER BY rank;

Run this after opening the example shop and clicking around for a minute, and you'll get a few rows of numbers that mean almost nothing. Which is the correct result. The figures in the walkthrough come from a hundred or so sessions of the same fairly artificial clicking about. That's enough for the shape to become visible: around 47% of shown results at rank one were clicked, 9% at rank two, 1% at rank three, and nothing below that.

I'm not suggesting those numbers say anything general about search behaviour. They say something about our small demo dataset 😄 What they do demonstrate is why ordinal is attached to every impression and click; You can't interpret clicks without thinking about position. A result at rank one and exactly the same result at rank ten do not have the same opportunity to be clicked. Which leads to the other important boundary in this release.

Collection, not judgment​

This is the collection half. Today, you can use it to find out:

• what people actually search for
• which queries return results that nobody touches
• where users reformulate
• which searches eventually lead to conversions; and
• which queries probably belong in your evaluation sets.

That last one is useful before doing anything clever with click models. The queries your users actually care about are rarely the twenty queries somebody happened to write down when search was first built. If a query is common, commercially important, or repeatedly going badly in production, it probably deserves to be represented in your relevance evaluation.

Today, copying those queries out of the Insights workspace and into a query set is manual. What Releval does not do yet is turn those interactions into relevance judgments automatically. That's intentional because a click is evidence of relevance, but that isn't the same thing as a relevance judgment. Position bias is the obvious example here; Results at the top get clicked more because they're at the top. You can't divide clicks by impressions, call the result a relevance grade, and pretend you've removed that effect. There is a reason click models have generated quite a lot of information retrieval research. That's also where I think this gets more interesting. A later version of Releval may use click models over the collected UBI data to derive implicit judgments that can feed back into evaluations. The important part is doing that in a way that accounts for the biases in the interaction data rather than treating every click as ground truth.

The tracker is the first piece necessary in this journey: collect the raw behaviour, preserve enough context to model it properly, and make the data available for analysis.

Where to get it​

With a bundler:

npm install @releval/tracker

If you don't use a bundler, grab releval-tracker.global.js from the latest release and serve it with your own static assets. It exposes a global Releval.

There's also a React adapter under @releval/tracker/react, with the SearchResults context and impression hook, so you don't need to write another IntersectionObserver wrapper yourself.

The browser tracker guide has the full reference, and there are two example applications in the repository if you want to see the integration end to end.

Happy tracking! 🎉

"You can't improve what you don't measure."

Peter Drucker, allegedly

I know. This quote gets wheeled out for everything from sales targets to step counts. Hear me out though. Search relevance is one of the places where I think it earns its keep.

I've watched a similar story play out on plenty of search teams over the years.

Someone makes a ranking change that looks obviously better than what was there before:

• A synonym list fixes a product category people kept complaining about.
• A recency boost pushes newer content higher.
• Some additional leaf queries improve precision for a handful of awkward cases.

The change fixes the queries someone happened to type while testing it. The before and after screenshots look good in the pull request, so it ships.

A few weeks later, someone notices search has quietly become worse somewhere else:

• The synonym list changed the meaning of queries in another category.
• The recency boost buried useful evergreen content.
• The extra leaf queries hurt recall and increased the number of zero-result searches.

Search relevance is still one of those areas where changes can be shipped with surprisingly little evidence.

Nobody would merge a performance change because the application felt faster. You'd want a benchmark, some measurements, and ideally a regression test. Ranking changes quite often get a handful of manually inspected queries and a "looks good to me".

Changing ranking isn't particularly difficult. Knowing what the change did to all the queries you didn't test is the harder part. The regressions usually get found eventually. They just get found through users, support tickets, analytics, or not at all when users get poor results and go elsewhere. And it generally isn't because the team doesn't care. The teams I've seen run into this problem have been good teams. They just haven't had a repeatable way to answer a fairly basic question:

"Was this actually a good change overall?"

Releval is my attempt to build the tooling I've wanted when answering that question, and today I'm releasing Releval 1.0.

Releval is a platform for evaluating, tracking and improving search relevance.

You point it at a search system, give it a representative set of queries and some relevance judgments, and it runs those queries, records the results and calculates ranking metrics. Change something, run the same evaluation again, and compare what moved. That's more or less the idea.

Most of the platform exists to make that loop cheap and repeatable enough that it can become part of how ranking changes are developed rather than something done occasionally after the fact.

The loop​

Everything starts with a search endpoint: the system you want to measure.

Releval doesn't particularly care what's behind it. Elasticsearch, OpenSearch, Solr, Vespa, a custom search API, or something else accessible over HTTP can all be evaluated.

Then you need three things:

1. A query set containing the queries you care about.
2. A query template describing how those queries should be sent to the search endpoint.
3. Relevance judgments describing what "good" looks like for each query.

Run an evaluation and Releval executes each query, stores the returned candidates and calculates metrics at the run, query and candidate level.

The metrics are the usual information retrieval ones: precision, recall, F-score, MAP, MRR, DCG, NDCG and ERR, calculated at whatever depth makes sense for your search experience. If users rarely look beyond the first ten results, measuring the first thousand probably isn't telling you much.

Judgments can be binary or graded, using scales from 0–4 or 0–9. Coarser scales are faster and easier to judge against; finer scales preserve more distinction between results. My preference is to use the coarsest scale that captures differences you actually care about.

A single evaluation run is useful, but comparison is where things become much more interesting. Make a change like:

• Change the analyser.
• Adjust a boost.
• Add a reranker.
• Replace the search engine entirely.

Then run the same evaluation again.

Each run keeps its configuration, results and metrics, so instead of asking:

Does search feel better?

you can ask:

Which metrics moved? Which queries improved? Which got worse? Which results changed?

Those are much better questions to be answering in a pull request, moving to an A/B test, or before shipping the change.

Judgments are the expensive bit​

There isn't really a way around this: ranking metrics need relevance judgments. Somebody, or something, has to look at a result for a query and decide how relevant it is. Those judgments are some of the most valuable data you can have about a search system, but producing them can also be tedious and expensive. Releval helps to reduce that cost without pretending it doesn't exist.

You can judge results manually using a keyboard-driven interface designed for working through a judgment backlog quickly. You can import grades you already have as a judgment list, whether they came from click data, a previous annotation exercise, another evaluation system, or a spreadsheet somebody has been maintaining. You can also use an LLM-as-a-judge for results that haven't been graded yet. Releval supports OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Ollama and OpenAI-compatible endpoints.

These sources are intended to work together rather than being mutually exclusive. For the same query and result, Releval prefers a deliberate human judgment over an AI-generated judgment. Existing data can provide coverage without allowing a lower-quality source to silently replace a better one.

AI judging is useful here, but I don't think it's useful to pretend that it solves relevance judgment entirely. Its main advantage is coverage. If you have two thousand queries, having a model make a first pass can be much more practical than manually judging every result before you can calculate anything. You can then spend human attention on the queries that are important, ambiguous or where the model appears to get things wrong. That's generally the way I'd use it.

Judgments also carry forward between runs. Once a candidate has been judged for a query, Releval can reuse that judgment rather than asking you to grade the same result every time you adjust a boost.

The headline number isn't enough​

Suppose overall NDCG goes up. That's useful to know, but it isn't enough to tell you whether the change was good. Maybe most queries improved slightly while several important ones regressed badly. Or perhaps a large number of low-value queries improved and masked a regression in the small group of queries that matter most to the business.

Two implementations can have very similar aggregate scores while behaving quite differently query by query. Metrics tell you where to look. You still need to understand what changed. Releval provides a few different ways to do that.

Query Lab is intended for the fast development loop. Run a query, change the template, inspect the results, add judgments and compare against a baseline. Once you're happy with the behaviour for the queries you're working on, run the full evaluation.

I think of Query Lab as the workbench and an evaluation run as the regression test.

The dashboard tracks metrics against previous runs and over time. It also records where the evaluation configuration changed. This is useful because if the metrics suddenly move, you want to know whether the search implementation changed or whether somebody changed the query set, judgments or evaluation configuration.

You can also compare two runs head to head: aggregate metrics, per-query deltas and the individual candidates that moved. That makes it possible to go from "the average improved" to understanding where that improvement came from, what regressed, and whether the ranking changed in the way you intended.

Sometimes the right result is "nothing changed"​

Not every search change is intended to affect relevance. You might be migrating from Solr to Elasticsearch, replacing an API, introducing a cache, upgrading a search engine, or changing infrastructure. In those cases, the goal may be that users see exactly the same results.

Releval can compare ranked result lists directly using similarity metrics such as Rank-Biased Overlap, without requiring relevance judgments. That lets you distinguish between changing the implementation and changing what users actually see. During a migration, that's a useful distinction to be able to measure rather than assume.

Put it in the pipeline​

Everything available through the UI is also available through APIs. Releval has been built from the beginning with automation in mind.

App clients provide machine-to-machine access for CI and other automation, and there's an MCP server for using Releval through tools and AI assistants.

The important bit for me is making relevance evaluation easy enough to run as part of normal development. Software testing became ubiquitous partly because we made it cheap and routine. You change some code, tests run, and you find out whether behaviour you care about changed. I'd like ranking changes to work much the same way.

A pull request changes ranking behaviour. An evaluation runs against an accepted baseline. You can see which metrics moved and which queries regressed before the change reaches production. It doesn't need to be a special relevance exercise or a weekly review. It can just be another check associated with changing the system.

You're already testing in production​

Every query your users run is effectively another test case. The difference is that you probably didn't choose it, and you may not be looking at the result.

Releval has integrated User Behavior Insights for capturing queries and subsequent behaviour using the open UBI standard: clicks, abandonment, reformulations and other search interactions. This data is useful in two ways:

• As a source of implicit relevance information.

and arguably more useful,

• As a means to discovering which queries actually deserve to be in your evaluation set.

The queries your users care about are rarely identical to the examples somebody came up with while building the search system. User Behavior Insights therefore gives you a feedback loop:

Observe what people search for. Find important or poorly performing queries. Add them to an evaluation. Change the ranking. Measure the result. Keep watching.

Over time, the evaluation set becomes less of a static test fixture and more of a record of the search problems you care about.

Running it​

Releval runs in your own infrastructure, so queries, results and judgments don't need to leave your environment.

The quickest way to get started is Docker Compose. The Getting started guide covers installation and running a first evaluation.

If you'd rather watch than read, the video below goes through the complete flow: connecting a search system, creating a query set, evaluating two query templates, generating judgments with an AI judge, and comparing the runs.

That's 1.0​

Releval is the platform I wanted when I've had to work out whether a ranking change actually made search better. Ranking changes deserve the same kind of evidence we'd expect when changing other important application behaviour. Releval is my attempt to make collecting that evidence straightforward enough that it can happen on every change.

The introduction covers the main concepts, Getting started will take you through a first evaluation, and the community repository is the place to report problems, suggest improvements and discuss where the platform should go next.

I'd particularly like feedback from people building and operating real search systems. If something doesn't fit the way you work, I'd like to hear about it. Those gaps are generally more useful than hearing that everything worked as expected.

Give Releval a try, and let me know how you get on. 😎