Judge Types
Score your evals with LLM judges, or with fast deterministic checks that cost nothing to run
A judge is a method of running an eval: it takes the output of a task run, and produces scores. Every eval in Kiln can have many judges, and each judge has a type.
Kiln offers two families of judge types:
LLM Judges: a model reads the output and grades it against criteria you write. Best for subjective qualities like tone, helpfulness, toxicity, etc.
Programmatic Judges: code inspects the output or the trace and returns a pass/fail. No model call, so they're fast, free, and return the same answer every time. Best for anything you can state as a rule.
Prefer a programmatic judge when one fits.
An LLM judge that decides "did the agent call get_weather before answering?" costs money on every run and can change its mind. A Tool Call Check answers the same question for free, identically, every time.
Save LLM judges for the questions that genuinely require subjective judgement.
The Judge Types
Programmatic Judge
The agent called the right tools, in the right order, with the right arguments
To add a judge, open your eval and create a new judge, or pick a type directly from the "Select an Eval Type" screen when creating a new eval. Programmatic judges are listed under the "Programmatic Judges" heading.
Choosing a Judge Type
Work down this list and stop at the first one that fits:
Is there exactly one right answer? Use Exact Match (or Set Check if the answer is a set of values).
Can you state the rule as text matching? Use Contains or Pattern Match. Good for format rules ("always ends with a citation", "never mentions a competitor").
Is it about what the agent did, not what it said? Use Tool Call Check for which tools ran, or Step Count Check for how many steps it took.
Is the rule real, but too complex for the above? Use a Code judge. It can also call an LLM for the subjective part, so you can filter a huge trace down in code and only pay for a judgement on what's left. See Code Judges.
Otherwise, use an LLM as Judge. Subjective quality needs a model.
Output to Check
The Exact Match, Pattern Match, Contains and Set Check types ask which part of the run they should look at:
Final Message: the model's final output. The default, and what you want most of the time.
Entire Trace: the whole conversation, including tool calls, as JSON.
Custom (Jinja): extract part of the output or trace using a Jinja expression.
Custom expressions are useful for structured outputs and agent traces. Some examples:
Extract a field from JSON output
(final_message | fromjson).user.status
Truncate a long output
final_message | truncate(200)
Last message in the trace
trace[-1].content
Count messages in the trace
trace | length
Name of a tool called in the trace
trace[-1].tool_calls[0].function.name
Tool Call Check and Step Count Check always read the trace, so they don't offer this option. Code judges receive the output and trace directly, and pick out what they need in Python.
LLM as Judge
A model reads the output and grades it against criteria you write, producing a score for each of your eval's output scores. You choose the judge model and provider, write the judge prompt and evaluation instructions, and can optionally enable G-Eval for more nuanced scores.
See our LLM Judges guide for each of these options.
Programmatic Judges
Exact Match
Passes when the output equals an expected value.
Expected Value: the value the output should equal.
Case Sensitive: on by default.
Output to Check: see above.
Best for tasks with a single correct answer: classification labels, yes/no answers, extracted IDs.
Pattern Match
Passes when the output matches a regular expression — or when it doesn't, if you set the mode to "must not match".
Expected Pattern (Regex): any Python regular expression.
Match Mode: must match, or must not match.
Output to Check: see above.
Best for format rules. A "must not match" pattern is a cheap way to catch an output that keeps leaking something it shouldn't, like a placeholder string or an internal ID format.
Contains
Passes when the output contains a substring — or when it doesn't, if you set the mode to "must not contain".
Expected Substring: the text to look for.
Case Sensitive: on by default.
Match Mode: must contain, or must not contain.
Output to Check: see above.
Simpler than Pattern Match, and usually clearer to a teammate reading your eval later. Reach for Pattern Match only when a plain substring won't do.
Set Check
Parses a set of values from the output and compares it to an expected set.
Expected Values: the values to compare against.
Comparison Mode:
subset(everything found must be in the expected set),superset(everything expected must be found), orequal(exactly the same values).Output to Check: see above.
Best for multi-label classification, tag extraction, or any task where the output is a list and the order doesn't matter.
Tool Call Check
Inspects the agent's trace to check it called the tools you expected.
Expected Tools: one or more tools, each optionally with expected arguments. Each argument can be matched exactly, by substring, or by regular expression.
Match Mode:
All (any order): every expected tool was called
Any: at least one expected tool was called
Ordered (in list order): the expected tools were called, in the order listed
Never: none of the listed tools were called
Unlisted Tool Calls: allow any other tool the agent calls, or fail the check if it calls anything not on your list. Not shown when Match Mode is "Never".
This is the deterministic way to test tool use. It answers "did the agent do the right thing?" for free, where an LLM judge would need to read the whole trace and form an opinion. See Evaluate Appropriate Tool Use for the full workflow, including when you still want an LLM judge.
Step Count Check
Counts steps in the agent's trace and passes when the count is within bounds you set.
What to Count: tool calls, model responses, or conversation turns.
Bounds: a Minimum, a Maximum, or both — at least one is required.
Best for agent efficiency. Cap an agent at 5 tool calls to catch runaway loops, or require at least 1 to confirm it actually used a tool instead of answering from memory.
Code (Beta)
Write a custom Python score() function. It can read the output, the full trace, and the task input, and it can call other tools — including built-in tools that call an LLM.
See the Code Judges guide for details.
Judges and Human Ratings
Kiln's judge comparison tools measure how closely a judge's scores match human ratings from your golden dataset. This is essential for LLM judges: an LLM judge is an approximation of human preference, and you need to know how good the approximation is.
Programmatic judges are a different kind of thing. They don't approximate anything — a Pattern Match either encodes the rule you meant or it doesn't, and it will return the same answer forever. You generally don't need a golden set or human ratings to trust one.
Running judge comparison on a programmatic judge is still allowed, and there's one case where it's genuinely useful: confirming that the rule you wrote actually captures what humans care about. If your Tool Call Check passes items your subject matter expert would fail, the check is wrong, not the human — and comparison will show you that.
If your eval only uses programmatic judges, you can skip the golden dataset and human rating steps entirely, and go straight to comparing run methods.
Last updated