Prompt templates

Each AI judge has a required Handlebars prompt template that defines what the model is asked, and the shape of the response it must return. The UI pre-fills a sensible default when you create a judge, which you can take as-is or adapt to your domain.

A judge on a TypeSafe provider uses Jev, whose template renders a question rather than a prompt; see Jev question templates.

Available variables

The Handlebars context exposes the following variables:

Variable	Type	Description
`query`	string \| object	The query value from the query set. A string for plain text query sets, or the JSON object for structured query sets.
`candidates`	array	Candidates being judged in this prompt. Length is 1 unless `batch_size > 1`.
`candidates[].id`	string	The candidate's ID. Must be referenced in the response so grades can be matched back to candidates.
`candidates[].title`	string	The candidate's title (from candidates mapping).
`candidates[].fields`	object	Additional mapped fields (price, brand, description, etc.). May be absent.
`candidates[].image`	string	The candidate's image. Only present when `include_images: true` and the candidate has a mapped image.
`scale`	string	The evaluation scale: `binary`, `graded` or `detailed`.
`scale_min`	number	The lowest grade the evaluation scale accepts.
`scale_max`	number	The highest grade the scale accepts.
`scale_labels`	array	Each entry has `grade` (number) and `label` (human-readable name like "Highly Relevant").
`evaluation_name`	string	The parent evaluation's name. Useful for context-setting prompts.
`evaluation_description`	string	The parent evaluation's description, if set.

Standard Handlebars syntax is supported, including {{#each}}, {{#if}}, {{else}}, and the helpers Releval registers for query templates. The eq helper compares two values, so a template can vary with the scale: {{#if (eq scale "detailed")}}.

When to customise the default

The pre-filled template works well for most domains. Adapt it when you need to:

Inject domain-specific guidelines (e.g. "treat refurbished items as Marginally Relevant").
Constrain the model to a stricter rubric.
Localise the prompt (the default is English-only).
Emphasise particular fields (e.g. brand, category) that appear in candidates[].fields.

Required response format

Whatever your template asks, the model's response must contain one <candidate> block per candidate sent in the prompt:

<candidate id="abc123">
<reasoning>
The query asked for "running shoes" and this candidate is a hiking boot.
The product description does not mention running.
</reasoning>
<grade>
1
</grade>
</candidate>

Three requirements apply. If any fails, the affected candidates are not judged:

Every candidate ID sent in the prompt must appear in exactly one <candidate> block.
Each <grade> must be an integer.
Each grade must be within the evaluation scale's range (e.g. 0–4 for graded).

The <reasoning> block is optional but strongly recommended: it is stored on the judgment and shown in the UI alongside the grade.

Info

The response must contain <candidate id="...">…</candidate> blocks with <grade> and <reasoning> children. Don't ask the model to return JSON or Markdown tables.

A worked custom template

Here's a custom template that adds an e-commerce rubric and explicitly asks the model to penalise out-of-stock items:

You are evaluating product search relevance for an online grocery store.

The shopper searched for: **{{query}}**

Rate each candidate on this scale:
{{#each scale_labels}}
- {{this.grade}} ({{this.label}})
{{/each}}

Apply these rules:
- Exact category and intent match → top of scale.
- Acceptable substitute (e.g. shopper searched "milk", saw "oat milk") → middle.
- Same product type, wrong attribute (e.g. "whole milk" → "skim milk") → middle-low.
- Out of stock → drop one grade.
- Off-topic → 0.

Candidates to judge:
{{#each candidates}}
- ID `{{this.id}}`: {{this.title}}{{#if this.fields.price}} (£{{this.fields.price}}){{/if}}{{#if this.fields.in_stock}}{{else}} [OUT OF STOCK]{{/if}}
{{/each}}

For each candidate, return:

{{#each candidates}}
<candidate id="{{this.id}}">
<reasoning>One sentence on why.</reasoning>
<grade>{{../scale_min}}-{{../scale_max}}</grade>
</candidate>
{{/each}}

Jev question templates

A judge on a TypeSafe provider grades with Jev, which does not read a prompt or write a reply. Jev answers a typed question: given some content, it returns a probability for each answer you define. So a Jev judge's template is not a prompt. It renders the question Jev is asked about each result: what to judge, and what each grade means.

The template renders a JSON object with exactly two properties:

instructions: the question, as a string, or as an object or array when it carries extra context.
criteria: one description per grade, lowest grade first.

Releval builds the rest of each request. It sends one request per result, with the query and that result as the content being judged:

{
  "model": "jev-latest",
  "state": {
    "query": "star wars",
    "result": {
      "title": "The Empire Strikes Back",
      "fields": { "year": 1980, "genre": "Sci-Fi" }
    }
  },
  "questions": {
    "relevance": {
      "type": "score",
      "instructions": "How well does `result` match what the searcher is looking for with `query`?",
      "criteria": ["...one description per grade..."]
    }
  }
}

The query keeps its structure when the query set has several parameters, and fields holds the result's mapped fields. Refer to them in the instructions by path, in backticks: `query`, `result`, or `result.fields.price`.

Variables

The template renders once per judging run, not once per result, so it sees the scale and the evaluation but never the query or the results:

Variable	Description
`scale`	The evaluation run's scale: `binary`, `graded` or `detailed`.
`scale_min`, `scale_max`	The lowest and highest grades on the scale.
`scale_labels`	Each entry has `grade` and `label`.
`evaluation_name`, `evaluation_description`	The parent evaluation's name and description.

Interpolated values are JSON-escaped, so "{{evaluation_description}}" is safe inside a JSON string.

One criterion per grade, for every scale

The evaluation run picks the scale, so a Jev judge must describe the grades of all three: 2 criteria for binary, 5 for graded and 10 for detailed. Branch on scale with the eq helper. Releval renders the template for every scale when you save the judge, and reports any scale whose criteria are the wrong length or whose output is not valid JSON.

The default template, shortened to the binary scale:

{
  "instructions": "How well does `result` match what the searcher is looking for with `query`?",
  "criteria": [
    {{#if (eq scale "binary")}}
    "The result does not help with what the query is looking for",
    "The result helps with what the query is looking for"
    {{/if}}
    {{#if (eq scale "graded")}}
    ...five descriptions...
    {{/if}}
    {{#if (eq scale "detailed")}}
    ...ten descriptions...
    {{/if}}
  ]
}

Writing criteria

Describe situations, not degrees. Jev judges each grade on its own against the result, without seeing the neighbouring grades, so "Moderately relevant" gives it nothing to match. "Covers part of what the query is looking for but misses something important" does. Criteria written as degrees spread the probability across grades and show up as low confidence.

When Jev keeps splitting between two neighbouring grades on results you think are clear, give each grade an object with the same fields, and add a few examples that look like your data:

"criteria": [
  { "what": "Unrelated to the query", "examples": ["a cookbook for the query 'winter tyres'"] },
  { "what": "Exactly what the query asks for", "examples": ["winter tyres in the size searched for"] }
]

To give Jev context about the search, put it in the instructions as an object with the question in one field:

"instructions": {
  "search": "{{evaluation_description}}",
  "question": "How well does `result` match what someone searching `search` wants from `query`?"
},

Writing JSON in Handlebars

Keep each scale's criteria inside its own {{#if}} block, so no comma is left dangling between blocks.
Never put { directly before {{: Handlebars reads {{{ as its triple-stash syntax. Put a space or a line break between them.
A template for a chat model is not valid for a Jev judge, and the other way around. Relinking a judge from one kind of provider to the other asks for a template of the new kind.

Iterating on a template

The prompt template lives on the judge, so changes apply to all future runs that use it. To test a template change without burning credits on a full run:

Clone the judge (or create a sibling using a free local provider such as ollama).
Run it on a small evaluation run (10–20 queries).
Inspect the per-candidate reasoning in the UI, or for a Jev judge the confidence and the probability on each grade.
Refine the template, repeat.
Once stable, point your full evaluations at the production judge.

When you're ready, see Running an AI judging run to start judging.

Available variables​

When to customise the default​

Required response format​

A worked custom template​

Jev question templates​

Variables​

One criterion per grade, for every scale​

Writing criteria​

Writing JSON in Handlebars​

Iterating on a template​