Skip to main content

Running an AI judging run

Once you've created and tested an AI judge, you can use it to judge any completed evaluation run. An AI judging run is the unit of work: one judge applied to one evaluation run, executed asynchronously.

Prerequisites​

The target evaluation run must satisfy two conditions:

  • The run must be completed. AI judging cannot start on a run that is still pending, queued, running, or failed; wait for the run to finish first.
  • The run must not be locked. A locked run rejects all judgment changes, including those produced by an AI judging run.

Starting a judging run​

In the UI​

  1. Open the Evaluation Run detail page.
  2. Click Run AI Judge in the sidebar.
  3. Choose the AI Judge to use from the dropdown.
  4. Click Start.

The run is queued immediately and a progress indicator appears as candidates are graded.

Using the API​

curl -X POST "https://${RELEVAL_HOST}/api/v1/evaluations/runs/${RUN_ID}/ai-judge" \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '{
    "ai_judge_id": "${AI_JUDGE_ID}"
  }'

The response contains the newly created judging run, including its identifier and an initial status of queued.

Multiple AI judging runs can target the same evaluation run, for example to compare grades from two different models. Later runs overwrite earlier judgments for the same candidate; if you need to keep both sets, export the first set's judgments before starting the second.

Monitoring progress​

Each AI judging run reports overall progress, how many candidates have been judged successfully, how many failed, and the running input/output token totals across all LLM calls so far. The UI updates these live as the run progresses. To monitor from your own tooling, list the AI judging runs for an evaluation run and re-fetch on a backoff until the status settles:

curl "https://${RELEVAL_HOST}/api/v1/evaluations/runs/${RUN_ID}/ai-judging-runs" \
  -H "Authorization: Bearer ${TOKEN}"

Stop polling once the status is no longer queued or running.

Cancelling a run​

A queued or running AI judging run can be cancelled at any time:

curl -X POST "https://${RELEVAL_HOST}/api/v1/ai-judging-runs/${JUDGING_RUN_ID}/cancel" \
  -H "Authorization: Bearer ${TOKEN}"

Cancellation marks the run as cancelled and stops further LLM calls. Calls that were already in flight may still complete and write their judgments, leaving the run in a "partially judged" state: usable, but not exhaustive. Start a new judging run to fill in the gaps if needed.

Cancellation is the right tool when:

  • You realise you picked the wrong judge or model.
  • The run is consuming more tokens than expected.
  • The provider is rate-limiting heavily and you want to retry later with a lower concurrency.

Handling failures​

Two kinds of failure can occur:

  • Per-candidate failure: the provider returned a response Releval could not parse, or it returned a grade outside the evaluation scale's range. The candidate is counted as failed, no judgment is written, and the rest of the run continues.
  • Run failure: an unrecoverable error such as authentication failure or repeated provider outage. The run moves to a failed status with an error message describing the cause.

Common causes of per-candidate failures:

  • The model returned JSON or Markdown instead of <candidate> blocks. Refine the prompt template.
  • The model returned a grade outside the scale (e.g. 5 on a 0–4 graded scale). Make the scale range explicit in the prompt.
  • The model returned grades for fewer candidates than were sent. Lower the batch size until it reliably handles every candidate in a batch.

For a judge on a TypeSafe provider, there is no reply to parse. A candidate fails only when Jev could not be reached or kept refusing the request after retries, for example while rate limited; a rejected API key fails every candidate, and so the run.

For run failures, fix the underlying issue (rotate the API key, raise the provider quota, adjust the endpoint URL) and start a new judging run on the same evaluation run.

Reviewing AI-generated judgments​

AI judgments appear alongside human ones in the judging UI, with the AI judge's name shown as the judge of record. Each judgment carries the <reasoning> text the model returned, which is invaluable for spot-checking quality.

Judgments from a Jev judge​

Jev gives no reasoning. It puts a probability on every grade, and Releval records the grade as the probability-weighted average of the grades, rounded to the nearest one. Each Jev judgment shows that distribution as a bar per grade, with the grade given highlighted, and Jev's confidence: 1 when all the probability is on one grade, falling as it spreads. Confidence describes how certain Jev was, not whether it was right, so check judgments that are confident but surprising as well as those with low confidence.

Low confidence usually means the grade descriptions overlap for that result, or the result does not say enough to place it. Results where Jev splits between two neighbouring grades are the ones to sample first, and a sign that those two criteria need sharper wording.

Jev reads a result as data, not as instructions, but a result written to argue for its own relevance can still pull its grade up. The same is true of any AI judge; sampling the judgments guards against it.

A reasonable workflow once a run completes:

  1. Sort by candidates the AI marked highly relevant and confirm a few. Hallucinated relevance is the most damaging error.
  2. Sort by candidates marked not relevant at the top of the result list. Genuine "wrong results" should appear here; if they don't, the prompt may be too lenient.
  3. Adjust the prompt template if needed and re-run on a small evaluation to verify the fix before re-running on full corpora.