Skip to main content

Query Lab

Query Lab is an interactive workbench for a single query. It sits on an evaluation as a tab alongside Runs and Dashboard, and it's where you dial in one query before committing a full run: send it to the evaluation's endpoint, see the candidates that come back, judge them, diff them against your last run, and - once it looks right - promote the configuration into a full evaluation run.

Where a full run executes your entire query set as a background job and computes metrics, Query Lab runs one query synchronously in the page. It's the fast iteration loop; the full run is the commit.

Opening Query Lab​

Open an evaluation and select the Query Lab tab. The evaluation must have a query set attached - the lab draws its queries from it.

Running a query​

  1. Choose a query. Either pick an entry from the evaluation's query set, or switch to Custom query and type your own. A custom query can be plain text or JSON (for example {"query": "shoes"}).
  2. (Optional) override the template. Expand template overrides to edit the request the query is rendered into - the query string, request body, and HTTP headers, as Handlebars templates. They start pre-filled from the evaluation's attached query template; edits here apply to the lab session only (and, if you promote, to the run you create).
  3. Run the query. Releval renders the template, sends the request to the endpoint, and maps the response using the endpoint's candidates mapping - exactly as a real run would.

The results panel shows the HTTP status, the resolved request URL, and the rendered request body, followed by the candidates the endpoint returned.

Judging in the lab​

Pick a scale (binary, graded, or detailed) and rate candidates inline. Ratings are saved as judgments against the endpoint's corpus, keyed to the query - so the same judgments are available to any run over that query, and switching scales keeps each scale's ratings.

Comparing against a baseline​

If the evaluation has a completed or locked run, the results panel diffs the lab's candidates against that baseline run: each candidate shows how its position moved (+n / -n), new candidates are badged NEW, and a footer notes how many baseline candidates dropped out. It's the quickest way to see whether a template tweak actually changed the ranking before you spend a full run on it.

Promoting to a full run​

Create Full Run turns the lab session into a real evaluation run. You give it a name, scale, and metrics (and optional tags); your template overrides carry across. The new run covers the whole query set, not just the query you were testing, and it's created in the Pending state - start it when you're ready. See the Create Evaluation Run endpoint for the API.

Note

Query Lab is distinct from two similar-looking tools: endpoint testing checks connectivity and candidate mapping with a fixed request, and query-template testing renders a template without calling the endpoint. Query Lab does both, then adds candidates, judging, and the baseline diff, scoped to an evaluation.