> For the complete documentation index, see [llms.txt](https://docs.kiln.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.kiln.tech/docs/evals-and-specs/judge-types.md).

# Judge Types

Score your evals with LLM judges, or with fast deterministic checks that cost nothing to run

A judge is a method of running an eval: it takes the output of a task run, and produces scores. Every eval in Kiln can have many judges, and each judge has a type.

Kiln offers two families of judge types:

* **LLM Judges**: a model reads the output and grades it against criteria you write. Best for subjective qualities like tone, helpfulness, toxicity, etc.
* **Programmatic Judges**: code inspects the output or the trace and returns a pass/fail. No model call, so they're fast, free, and return the same answer every time. Best for anything you can state as a rule.

{% hint style="success" %}
**Prefer a programmatic judge when one fits.**

An LLM judge that decides "did the agent call `get_weather` before answering?" costs money on every run and can change its mind. A [Tool Call Check](#tool-call-check) answers the same question for free, identically, every time.

Save LLM judges for the questions that genuinely require subjective judgement.
{% endhint %}

### The Judge Types

| Judge Type                            | Family             | What it scores                                                                 |
| ------------------------------------- | ------------------ | ------------------------------------------------------------------------------ |
| [LLM as Judge](#llm-as-judge)         | LLM Judge          | A model grades the output against a rubric you write                           |
| [Code](#code-beta)                    | Programmatic Judge | A custom Python `score()` function you write                                   |
| [Tool Call Check](#tool-call-check)   | Programmatic Judge | The agent called the right tools, in the right order, with the right arguments |
| [Exact Match](#exact-match)           | Programmatic Judge | The output equals an expected value                                            |
| [Pattern Match](#pattern-match)       | Programmatic Judge | The output matches (or doesn't match) a regular expression                     |
| [Contains](#contains)                 | Programmatic Judge | The output contains (or omits) a substring                                     |
| [Set Check](#set-check)               | Programmatic Judge | A set of values from the output matches an expected set                        |
| [Step Count Check](#step-count-check) | Programmatic Judge | The agent finished within an expected number of steps                          |

To add a judge, open your eval and create a new judge, or pick a type directly from the "Select an Eval Type" screen when creating a new eval. Programmatic judges are listed under the "Programmatic Judges" heading.

### Choosing a Judge Type

Work down this list and stop at the first one that fits:

1. **Is there exactly one right answer?** Use [Exact Match](#exact-match) (or [Set Check](#set-check) if the answer is a set of values).
2. **Can you state the rule as text matching?** Use [Contains](#contains) or [Pattern Match](#pattern-match). Good for format rules ("always ends with a citation", "never mentions a competitor").
3. **Is it about what the agent did, not what it said?** Use [Tool Call Check](#tool-call-check) for which tools ran, or [Step Count Check](#step-count-check) for how many steps it took.
4. **Is the rule real, but too complex for the above?** Use a [Code](#code-beta) judge. It can also call an LLM for the subjective part, so you can filter a huge trace down in code and only pay for a judgement on what's left. See [Code Judges](/docs/evals-and-specs/code-judges.md).
5. **Otherwise, use an** [**LLM as Judge**](#llm-as-judge). Subjective quality needs a model.

{% hint style="info" %}
A single eval can have judges of different types, and Kiln will show their scores side by side. A "helpfulness" eval can pair an LLM judge for tone with a Pattern Match that catches a formatting bug — you don't need to pick just one.
{% endhint %}

### Output to Check

The Exact Match, Pattern Match, Contains and Set Check types ask which part of the run they should look at:

* **Final Message**: the model's final output. The default, and what you want most of the time.
* **Entire Trace**: the whole conversation, including tool calls, as JSON.
* **Custom (Jinja)**: extract part of the output or trace using a Jinja expression.

Custom expressions are useful for structured outputs and agent traces. Some examples:

| Goal                               | Expression                                |
| ---------------------------------- | ----------------------------------------- |
| Extract a field from JSON output   | `(final_message \| fromjson).user.status` |
| Truncate a long output             | `final_message \| truncate(200)`          |
| Last message in the trace          | `trace[-1].content`                       |
| Count messages in the trace        | `trace \| length`                         |
| Name of a tool called in the trace | `trace[-1].tool_calls[0].function.name`   |

Tool Call Check and Step Count Check always read the trace, so they don't offer this option. [Code](#code-beta) judges receive the output and trace directly, and pick out what they need in Python.

### LLM as Judge

A model reads the output and grades it against criteria you write, producing a score for each of your eval's output scores. You choose the judge model and provider, write the judge prompt and evaluation instructions, and can optionally enable G-Eval for more nuanced scores.

See our [LLM Judges](/docs/evals-and-specs/llm-judges.md) guide for each of these options.

### Programmatic Judges

#### Exact Match

Passes when the output equals an expected value.

* **Expected Value**: the value the output should equal.
* **Case Sensitive**: on by default.
* **Output to Check**: see [above](#output-to-check).

Best for tasks with a single correct answer: classification labels, yes/no answers, extracted IDs.

#### Pattern Match

Passes when the output matches a regular expression — or when it doesn't, if you set the mode to "must not match".

* **Expected Pattern (Regex)**: any Python regular expression.
* **Match Mode**: must match, or must not match.
* **Output to Check**: see [above](#output-to-check).

Best for format rules. A "must not match" pattern is a cheap way to catch an output that keeps leaking something it shouldn't, like a placeholder string or an internal ID format.

#### Contains

Passes when the output contains a substring — or when it doesn't, if you set the mode to "must not contain".

* **Expected Substring**: the text to look for.
* **Case Sensitive**: on by default.
* **Match Mode**: must contain, or must not contain.
* **Output to Check**: see [above](#output-to-check).

Simpler than Pattern Match, and usually clearer to a teammate reading your eval later. Reach for Pattern Match only when a plain substring won't do.

#### Set Check

Parses a set of values from the output and compares it to an expected set.

* **Expected Values**: the values to compare against.
* **Comparison Mode**: `subset` (everything found must be in the expected set), `superset` (everything expected must be found), or `equal` (exactly the same values).
* **Output to Check**: see [above](#output-to-check).

Best for multi-label classification, tag extraction, or any task where the output is a list and the order doesn't matter.

#### Tool Call Check

Inspects the agent's trace to check it called the tools you expected.

* **Expected Tools**: one or more tools, each optionally with expected arguments. Each argument can be matched exactly, by substring, or by regular expression.
* **Match Mode**:
  * All (any order): every expected tool was called
  * Any: at least one expected tool was called
  * Ordered (in list order): the expected tools were called, in the order listed
  * Never: none of the listed tools were called
* **Unlisted Tool Calls**: allow any other tool the agent calls, or fail the check if it calls anything not on your list. Not shown when Match Mode is "Never".

This is the deterministic way to test tool use. It answers "did the agent do the right thing?" for free, where an LLM judge would need to read the whole trace and form an opinion. See [Evaluate Appropriate Tool Use](/docs/evals-and-specs/evaluate-appropriate-tool-use.md) for the full workflow, including when you still want an LLM judge.

#### Step Count Check

Counts steps in the agent's trace and passes when the count is within bounds you set.

* **What to Count**: tool calls, model responses, or conversation turns.
* **Bounds**: a **Minimum**, a **Maximum**, or both — at least one is required.

Best for agent efficiency. Cap an agent at 5 tool calls to catch runaway loops, or require at least 1 to confirm it actually used a tool instead of answering from memory.

#### Code (Beta)

Write a custom Python `score()` function. It can read the output, the full trace, and the task input, and it can call other tools — including built-in tools that call an LLM.

See the [Code Judges](/docs/evals-and-specs/code-judges.md) guide for details.

### Judges and Human Ratings

Kiln's [judge comparison](/docs/evals-and-specs/evaluations.md#finding-the-ideal-judge) tools measure how closely a judge's scores match human ratings from your golden dataset. This is essential for LLM judges: an LLM judge is an approximation of human preference, and you need to know how good the approximation is.

Programmatic judges are a different kind of thing. They don't approximate anything — a Pattern Match either encodes the rule you meant or it doesn't, and it will return the same answer forever. You generally don't need a golden set or human ratings to trust one.

Running judge comparison on a programmatic judge is still allowed, and there's one case where it's genuinely useful: confirming that the rule you wrote actually captures what humans care about. If your Tool Call Check passes items your subject matter expert would fail, the check is wrong, not the human — and comparison will show you that.

{% hint style="success" %}
If your eval only uses programmatic judges, you can skip the golden dataset and human rating steps entirely, and go straight to [comparing run methods](/docs/evals-and-specs/evaluations.md#finding-the-ideal-run-method).
{% endhint %}