> This page is also available as Markdown: append .md to its URL. The full index of Sikaru docs is at https://docs.sikaru.ai/llms.txt. Code examples use only operations from the public API reference at https://docs.sikaru.ai/api-reference.

# Graders

Facts on this page:

1. A Grader belongs to one agent. It asks yes/no questions about each conversation; a yes flags it. Verdicts: `flagged`, `looks_fine`, `inconclusive`.
2. Define one as `graders/<name>.toml` in the agent folder: `description`, `judge` (`"decision-model"`, `"llm"`, or `{ kind = "agent", agent = "<slug>" }`), `escalate` (decision model only), `[release_check]` `enabled` and `ceiling` (0 to 1), `[[criteria]]` `name` (optional) and `question`.
3. Version status: `provisional`, `live`, `retired`. A provisional version goes live once its agreement with your labels is at least 0.85 on at least 30 held-out labels and no worse than the current live version. Only the live version flags runs and checks releases.
4. Labels: `flagged` or `looks_fine` with an optional reason. Native CLI: `sikaru agent grader label NAME`.
5. Release check outcomes: passed, blocked, couldn't verify. Held publish refusals: `release_check_blocked`, `release_check_unverified`. Override: `sikaru agent publish --publication ID --anyway "<reason>"`, or `client.agent_documents.publish_anyway`.
6. Native CLI: `sikaru agent grader init|test|label|list|get|samples|run`. Generated API commands: `sikaru agent graders list|get|update|label|label_queue|samples|run|get_run|test|get_test|estimate_cost` (take `--agent-id`, the agent's ID). Python SDK: `client.graders.*`.
7. API base: `/v1/projects/{project_id}/agents/{agent_id}/graders`.

A **Grader** watches one agent's production conversations. It asks your yes/no questions about each conversation, and a yes flags it. You teach it by labeling conversations, and once it agrees with you it can check every new version of the agent before that version goes live.

```mermaid
flowchart LR
    File["graders/refunds.toml"] --> Provisional["Provisional version"]
    Labels["Your labels"] --> Provisional
    Provisional -->|"agrees with your labels"| Live["Live version"]
    Live --> Flags["Flags on runs"]
    Live --> Check["Release check"]
    Check -->|"passed"| Publish["New version goes live"]
    Check -->|"blocked or couldn't verify"| Held["Publish held"]
```

Each agent also starts with built-in Graders for common problems, such as made-up information, ignored instructions or a task left partly done. They need no file and score from the start.

Grading is off for personal agents and for agents whose end-user conversations are private.

## Define a Grader

A Grader is a file in the agent folder, `graders/<name>.toml`. Scaffold one with `sikaru agent grader init refunds` or `sikaru agent add grader/refunds`.

```toml
description = "Refunds issued when the customer asked for an exchange"
judge = "decision-model"   # or "llm", or { kind = "agent", agent = "billing-reviewer" }
escalate = false           # decision model only: send uncertain conversations to the LLM judge

[release_check]
enabled = true             # needs a live version of the Grader
ceiling = 0.1              # optional: block a release whose flag rate is above this

[[criteria]]
name = "refund-not-exchange"   # optional
question = "Does the agent refund when the customer asked for an exchange?"
```

| Field                   | Meaning                                                                        |
| ----------------------- | ------------------------------------------------------------------------------ |
| `description`           | What the Grader looks for. Up to 2,000 characters.                             |
| `judge`                 | Who answers the questions. See [Judges](#judges).                              |
| `escalate`              | Decision model only. Uncertain conversations go to the LLM judge.              |
| `release_check.enabled` | Check new versions of the agent before they go live.                           |
| `release_check.ceiling` | Optional flag rate, 0 to 1. A version flagged more often than this is blocked. |
| `criteria`              | Up to 50 yes/no questions. A yes flags the conversation.                       |

`sikaru agent push` and `sikaru agent publish` apply Grader files. An invalid file refuses the whole push and names its file and line. Changing only Grader files publishes no new agent version. Removing a file retires its Grader: it stops scoring and checking releases, and its labels stay. Adding the same name again restores it. See [Author and publish](/build/agents-and-releases) for the full folder rules.

## Judges

| Judge          | File value                             | Notes                                                               |
| -------------- | -------------------------------------- | ------------------------------------------------------------------- |
| Decision model | `"decision-model"`                     | The default. Can escalate uncertain conversations to the LLM judge. |
| LLM judge      | `"llm"`                                | A language model answers each question.                             |
| Agent judge    | `{ kind = "agent", agent = "<slug>" }` | One of your project's agents answers, using its live version.       |

LLM and agent judges use organization credits. Before switching, estimate the cost with `sikaru agent graders estimate_cost --project-id PROJECT --agent-id AGENT_ID --grader-id GRADER_ID --judge llm`.

Changing the judge or escalation makes a new provisional version. The live version keeps scoring until the new one clears the [promotion bar](#versions).

## Labels

Labels teach a Grader what you mean. Mark a conversation **Flagged** or **Looks Fine**, with an optional reason. Labeling a conversation again replaces its label. After you answer, you see what the Grader itself said.

```sh
sikaru agent grader label refunds
```

The command walks you through the conversations the Grader is least sure about, one at a time. Each shows the conversation's last steps without the Grader's verdict, so your answer stays your own.

New labels, edited questions and judge changes lead to a refit. While calibrating, Sikaru may add its own questions and weigh them; those never appear in your file, so a later push never undoes calibration.

## Versions

Every Grader version is fixed once made. Its status is one of:

| Status        | Meaning                                                                                                                  |
| ------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `provisional` | New, from an edit, a judge change or a refit. Its verdicts show beside the live ones but flag nothing and check nothing. |
| `live`        | The version that flags runs and runs release checks.                                                                     |
| `retired`     | Replaced by a newer live version.                                                                                        |

A provisional version goes live when its agreement with your labels is at least 0.85, measured on at least 30 labels it was not fitted on, and is no worse than the current live version's. `sikaru agent grader get refunds` shows each version's status, agreement and label counts.

A built-in Grader shows `uncalibrated` and scores until it is calibrated. A Grader you wrote shows `provisional` and flags nothing until its first version goes live.

## Verdicts and flags on runs

A Grader judges whole conversations. Each verdict is `flagged`, `looks_fine` or `inconclusive`. A run is flagged by a Grader when its conversation's latest verdict from the live version is `flagged`. The dashboard shows these flags on runs and lets you filter by them.

```sh
sikaru agent grader samples refunds
```

`samples` lists the Grader's recent verdicts, flagged conversations first, up to 50.

## Test a Grader

`sikaru agent grader test NAME` scores the local file on the agent's recent runs with the file's own judge. It saves no Grader and no verdict.

```sh
sikaru agent grader test refunds --last 20
```

| Exit code | Meaning                                                               |
| --------- | --------------------------------------------------------------------- |
| `0`       | Scored within the ceiling, or the file's release check is off.        |
| `1`       | The ceiling would block, the test failed, or the request was refused. |
| `2`       | Invalid input, sign-in needed, or no recent runs to score.            |

`sikaru agent grader run NAME` re-scores recent production runs with the Grader's live version, or its newest version when none is live. It waits and prints the flagged count. `--days` sets how far back to look, from 1 to 90 days (default 30).

## Release checks

A release check compares a new version of the agent with the live one before the new version goes live. It needs a live version of the Grader.

When you publish, Sikaru replays a set of cases on both versions and scores both with the Grader's live version. The cases are the conversations you labeled plus the conversations the Grader flagged in the last 30 days, up to 100.

| Outcome         | When                                                                                                       |
| --------------- | ---------------------------------------------------------------------------------------------------------- |
| Passed          | No regression, and the flag rate is within the ceiling.                                                    |
| Blocked         | The new version has clearly more conversations flagged only for it, or its flag rate is above the ceiling. |
| Couldn't verify | Too many cases were inconclusive, or too few were conclusive to decide.                                    |

With several checks, any block holds the release; otherwise any couldn't-verify holds it.

```sh
sikaru agent publish -p PROJECT
```

`publish` follows the checks until they settle. A held release prints what each Grader flagged and the exact command to release it anyway, then exits `1`. `--no-wait` returns while checks run. Over the API, a held publish answers with `release_check_blocked` or `release_check_unverified`, and the publication's status is `held`.

### Override a held release

If you have reviewed the flagged cases and still want the version live, publish it anyway with a reason:

```sh
sikaru agent publish -p PROJECT --publication PUBLICATION_ID --anyway "Flags are expected while the refund policy changes"
```

Sikaru records who released it, when and why. Publish Anyway is refused if the draft, the live version or the release check settings changed since the check ran; publish again instead.

You can also relax a check while it holds a release: turn it off or raise its ceiling. That needs a reason too, and is recorded the same way.

```sh
sikaru agent graders update --project-id PROJECT --agent-id AGENT_ID --grader-id GRADER_ID \
  --json '{"releaseCheck": {"enabled": true, "ceiling": 0.2}, "reason": "New policy raises expected flags"}'
```

## Permissions

| Action                                                                                     | Credentials                                                                                   |
| ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| List and read Graders, samples and cost estimates                                          | `runs:read` or `harness:read`                                                                 |
| Read conversation excerpts in the label queue                                              | `runs:read`                                                                                   |
| Test a file or re-score production runs                                                    | `managed_agents:write`                                                                        |
| Label, change the judge, escalation or release check, remove a Grader file, Publish Anyway | Grader admin: a signed-in member, or a project key created with Grader admin on the dashboard |

## CLI and API

Inside an agent folder, the `sikaru agent grader` commands take the Grader's file name and find the project and agent from the folder:

| Command                            | Does                                                |
| ---------------------------------- | --------------------------------------------------- |
| `sikaru agent grader init NAME`    | Write a starter `graders/NAME.toml`.                |
| `sikaru agent grader test NAME`    | Score the local file on recent runs; saves nothing. |
| `sikaru agent grader label NAME`   | Label conversations one at a time.                  |
| `sikaru agent grader list`         | List the agent's Graders, built-in ones included.   |
| `sikaru agent grader get NAME`     | Show one Grader, its versions and label counts.     |
| `sikaru agent grader samples NAME` | Recent verdicts, flagged first.                     |
| `sikaru agent grader run NAME`     | Re-score recent production runs.                    |

The API lives under `/v1/projects/{project_id}/agents/{agent_id}/graders` and takes the agent's ID. The same operations are on `client.graders` in the SDKs and under `sikaru agent graders` in the CLI: `list`, `get`, `update`, `label`, `label_queue`, `samples`, `run`, `get_run`, `test`, `get_test` and `estimate_cost`. Publish Anyway is `client.agent_documents.publish_anyway`.

```python
from sikaru_sdk import CheckChoice

client.graders.update(
    project_id="PROJECT", agent_id="AGENT_ID", grader_id="GRADER_ID",
    release_check=CheckChoice(enabled=True, ceiling=0.1),
)
```

## Next steps

#### [Author and publish](/build/agents-and-releases)

Add Grader files to the agent folder and publish.

#### [Evaluations](/improve/evaluations)

Apply a rubric to retained traces with an evaluation job.