Graders
Ask yes/no questions about every production conversation, teach a Grader with labels, and let its release check hold a version that makes things worse.
A Grader watches one agent’s production conversations. It asks your yes/no questions about each conversation, and a yes flags it. You teach it by labeling conversations, and once it agrees with you it can check every new version of the agent before that version goes live.
Each agent also starts with built-in Graders for common problems, such as made-up information, ignored instructions or a task left partly done. They need no file and score from the start.
Grading is off for personal agents and for agents whose end-user conversations are private.
Define a Grader
A Grader is a file in the agent folder, graders/<name>.toml. Scaffold one with sikaru agent grader init refunds or sikaru agent add grader/refunds.
sikaru agent push and sikaru agent publish apply Grader files. An invalid file refuses the whole push and names its file and line. Changing only Grader files publishes no new agent version. Removing a file retires its Grader: it stops scoring and checking releases, and its labels stay. Adding the same name again restores it. See Author and publish for the full folder rules.
Judges
LLM and agent judges use organization credits. Before switching, estimate the cost with sikaru agent graders estimate_cost --project-id PROJECT --agent-id AGENT_ID --grader-id GRADER_ID --judge llm.
Changing the judge or escalation makes a new provisional version. The live version keeps scoring until the new one clears the promotion bar.
Labels
Labels teach a Grader what you mean. Mark a conversation Flagged or Looks Fine, with an optional reason. Labeling a conversation again replaces its label. After you answer, you see what the Grader itself said.
The command walks you through the conversations the Grader is least sure about, one at a time. Each shows the conversation’s last steps without the Grader’s verdict, so your answer stays your own.
New labels, edited questions and judge changes lead to a refit. While calibrating, Sikaru may add its own questions and weigh them; those never appear in your file, so a later push never undoes calibration.
Versions
Every Grader version is fixed once made. Its status is one of:
A provisional version goes live when its agreement with your labels is at least 0.85, measured on at least 30 labels it was not fitted on, and is no worse than the current live version’s. sikaru agent grader get refunds shows each version’s status, agreement and label counts.
A built-in Grader shows uncalibrated and scores until it is calibrated. A Grader you wrote shows provisional and flags nothing until its first version goes live.
Verdicts and flags on runs
A Grader judges whole conversations. Each verdict is flagged, looks_fine or inconclusive. A run is flagged by a Grader when its conversation’s latest verdict from the live version is flagged. The dashboard shows these flags on runs and lets you filter by them.
samples lists the Grader’s recent verdicts, flagged conversations first, up to 50.
Test a Grader
sikaru agent grader test NAME scores the local file on the agent’s recent runs with the file’s own judge. It saves no Grader and no verdict.
sikaru agent grader run NAME re-scores recent production runs with the Grader’s live version, or its newest version when none is live. It waits and prints the flagged count. --days sets how far back to look, from 1 to 90 days (default 30).
Release checks
A release check compares a new version of the agent with the live one before the new version goes live. It needs a live version of the Grader.
When you publish, Sikaru replays a set of cases on both versions and scores both with the Grader’s live version. The cases are the conversations you labeled plus the conversations the Grader flagged in the last 30 days, up to 100.
With several checks, any block holds the release; otherwise any couldn’t-verify holds it.
publish follows the checks until they settle. A held release prints what each Grader flagged and the exact command to release it anyway, then exits 1. --no-wait returns while checks run. Over the API, a held publish answers with release_check_blocked or release_check_unverified, and the publication’s status is held.
Override a held release
If you have reviewed the flagged cases and still want the version live, publish it anyway with a reason:
Sikaru records who released it, when and why. Publish Anyway is refused if the draft, the live version or the release check settings changed since the check ran; publish again instead.
You can also relax a check while it holds a release: turn it off or raise its ceiling. That needs a reason too, and is recorded the same way.
Permissions
CLI and API
Inside an agent folder, the sikaru agent grader commands take the Grader’s file name and find the project and agent from the folder:
The API lives under /v1/projects/{project_id}/agents/{agent_id}/graders and takes the agent’s ID. The same operations are on client.graders in the SDKs and under sikaru agent graders in the CLI: list, get, update, label, label_queue, samples, run, get_run, test, get_test and estimate_cost. Publish Anyway is client.agent_documents.publish_anyway.
