Skip to main content

Judgment Lists

A Judgment List is a pre-existing set of relevance judgments that you upload to Releval. Unlike inline judgments that are made interactively during an evaluation run, judgment lists come from external sources such as:

  • Click-through logs or analytics
  • Previous evaluation campaigns
  • Crowdsourced relevance assessments
  • Expert annotations

Once uploaded, a list's grades feed straight into the evaluation metrics for matching queries on the same corpus, so you can score a run against judgments you already have instead of re-judging from scratch.

File format​

The judgment file follows the TREC qrels shape — qid iteration candidate_id grade — and comes in two forms, selected by the file extension.

Headered files​

JSON Lines (.jsonl / .json / .ndjson), CSV/TSV (.csv / .tsv), and Parquet (.parquet) files carry one judgment per row with named columns.

Required columns:

  • qid — the query id, a label grouping rows by query.
  • candidate_id — the id of the candidate being judged. It must equal the candidate id the endpoint returns for the grade to attach to a result.
  • query — the query, as text or a JSON query object. Releval matches grades to evaluation queries by hashing this, so it is required in headered files.
  • grade — an integer that must fall within the range of the scale you choose on upload: binary, graded, or detailed.

Optional columns:

  • iteration — accepted for qrels compatibility; ignored when scoring.
  • oid — an explicit query-object hash, overriding the value derived from query. Useful when round-tripping judgments exported from a Releval evaluation to preserve exact-query matching.
  • doc — a snapshot of the document text, useful when the originating system may not return the same document later.

JSON Lines example​

{ "qid": "q1", "query": "running shoes", "candidate_id": "sku-1001", "grade": 4 }
{ "qid": "q1", "query": "running shoes", "candidate_id": "sku-1002", "grade": 2 }
{ "qid": "q2", "query": "winter coat", "candidate_id": "sku-2050", "grade": 3 }

CSV example​

qid,candidate_id,query,grade
q1,sku-1001,running shoes,4
q1,sku-1002,running shoes,2
q2,sku-2050,winter coat,3

Raw qrels files​

Classic qrels files (.txt / .qrels) are accepted directly: headerless, whitespace-delimited, with four positional columns — qid iteration candidate_id grade.

q1 0 sku-1001 4
q1 0 sku-1002 2
q2 0 sku-2050 3

Raw qrels rows carry no query, so the qid is used verbatim as the match key. This lets judgments exported from Releval — whose qid is the query-text hash — round-trip as qrels and match on re-import. Because there is no query object to hash, raw qrels rows match corpora that use query-text (qid) match mode only; for query-object matching, upload a headered file with a query column.

Uploading a judgment list​

A judgment list belongs to a corpus, so create the corpus first if it doesn't already exist. On upload you also pick the type of the judgments (whether they are explicit ratings, implicit signals derived from behavior, golden examples, or crowdsourced) and the scale the grades use.

In the UI​

  1. Navigate to Judgment Lists and click Upload.
  2. Select the file to upload.
  3. Enter a Name, choose the Corpus it belongs to, the Type, and the Scale the grades use.
  4. Click Upload.

Using the API​

curl -X POST "https://${RELEVAL_HOST}/api/v1/judgment-lists/upload" \
-H "Authorization: Bearer ${TOKEN}" \
-F 'name=My Judgment List' \
-F 'corpus_id=${CORPUS_ID}' \
-F 'type=explicit' \
-F 'scale=graded' \
-F 'judgments=@judgments.jsonl'

How judgment lists feed evaluation metrics​

An uploaded judgment list contributes its grades to evaluation metrics automatically. There is no attach step: once a list is uploaded against a corpus, any evaluation run on that corpus picks up the grades for every query that matches, and its NDCG, MAP, Precision, and the other metrics are computed from them — exactly as if the grades had been entered by hand.

Matching follows the corpus's match mode: by default a row matches every evaluation query with the same query text. Two things have to line up for a grade to count — the row's grade must fall within the scale you pick on upload (out-of-range grades are rejected), and the candidate_id must equal the candidate id the endpoint returns, or the row has nothing to attach to.

When a query also has interactive or AI judgments​

Uploaded grades fill gaps; they never override curated judgments. For any one candidate the highest-priority source present wins, in this order:

  1. Human — an interactive judgment made on the Evaluate page.
  2. AI — a grade from an AI judging run.
  3. List — a grade from an uploaded judgment list.

So a candidate a person judged keeps the person's grade, while a candidate no one has judged inherits the uploaded grade. Deleting a list recomputes the affected metrics back to what they were without it.

Managing judgment lists​

List all judgment lists​

curl "https://${RELEVAL_HOST}/api/v1/judgment-lists" \
-H "Authorization: Bearer ${TOKEN}"

You can filter by endpoint type:

curl "https://${RELEVAL_HOST}/api/v1/judgment-lists?endpoint_type=elasticsearch" \
-H "Authorization: Bearer ${TOKEN}"

Delete judgment lists​

curl -X DELETE "https://${RELEVAL_HOST}/api/v1/judgment-lists?judgment_list_id=${ID}" \
-H "Authorization: Bearer ${TOKEN}"