Skip to navigation

Graders

Ask yes/no questions about every production conversation, teach a Grader with labels, and let its release check hold a version that makes things worse.

View as Markdown

A Grader watches one agent’s production conversations. It asks your yes/no questions about each conversation, and a yes flags it. You teach it by labeling conversations, and once it agrees with you it can check every new version of the agent before that version goes live.

agrees with your labels passed blocked or couldn't verify graders/refunds.toml Provisional version Your labels Live version Flags on runs Release check New version goes live Publish held

Each agent also starts with built-in Graders for common problems, such as made-up information, ignored instructions or a task left partly done. They need no file and score from the start.

Grading is off for personal agents and for agents whose end-user conversations are private.

Define a Grader

A Grader is a file in the agent folder, graders/<name>.toml. Scaffold one with sikaru agent grader init refunds or sikaru agent add grader/refunds.

description = "Refunds issued when the customer asked for an exchange"
judge = "decision-model" # or "llm", or { kind = "agent", agent = "billing-reviewer" }
escalate = false # decision model only: send uncertain conversations to the LLM judge
[release_check]
enabled = true # needs a live version of the Grader
ceiling = 0.1 # optional: block a release whose flag rate is above this
[[criteria]]
name = "refund-not-exchange" # optional
question = "Does the agent refund when the customer asked for an exchange?"
FieldMeaning
descriptionWhat the Grader looks for. Up to 2,000 characters.
judgeWho answers the questions. See Judges.
escalateDecision model only. Uncertain conversations go to the LLM judge.
release_check.enabledCheck new versions of the agent before they go live.
release_check.ceilingOptional flag rate, 0 to 1. A version flagged more often than this is blocked.
criteriaUp to 50 yes/no questions. A yes flags the conversation.

sikaru agent push and sikaru agent publish apply Grader files. An invalid file refuses the whole push and names its file and line. Changing only Grader files publishes no new agent version. Removing a file retires its Grader: it stops scoring and checking releases, and its labels stay. Adding the same name again restores it. See Author and publish for the full folder rules.

Judges

JudgeFile valueNotes
Decision model"decision-model"The default. Can escalate uncertain conversations to the LLM judge.
LLM judge"llm"A language model answers each question.
Agent judge{ kind = "agent", agent = "<slug>" }One of your project’s agents answers, using its live version.

LLM and agent judges use organization credits. Before switching, estimate the cost with sikaru agent graders estimate_cost --project-id PROJECT --agent-id AGENT_ID --grader-id GRADER_ID --judge llm.

Changing the judge or escalation makes a new provisional version. The live version keeps scoring until the new one clears the promotion bar.

Labels

Labels teach a Grader what you mean. Mark a conversation Flagged or Looks Fine, with an optional reason. Labeling a conversation again replaces its label. After you answer, you see what the Grader itself said.

sikaru agent grader label refunds

The command walks you through the conversations the Grader is least sure about, one at a time. Each shows the conversation’s last steps without the Grader’s verdict, so your answer stays your own.

New labels, edited questions and judge changes lead to a refit. While calibrating, Sikaru may add its own questions and weigh them; those never appear in your file, so a later push never undoes calibration.

Versions

Every Grader version is fixed once made. Its status is one of:

StatusMeaning
provisionalNew, from an edit, a judge change or a refit. Its verdicts show beside the live ones but flag nothing and check nothing.
liveThe version that flags runs and runs release checks.
retiredReplaced by a newer live version.

A provisional version goes live when its agreement with your labels is at least 0.85, measured on at least 30 labels it was not fitted on, and is no worse than the current live version’s. sikaru agent grader get refunds shows each version’s status, agreement and label counts.

A built-in Grader shows uncalibrated and scores until it is calibrated. A Grader you wrote shows provisional and flags nothing until its first version goes live.

Verdicts and flags on runs

A Grader judges whole conversations. Each verdict is flagged, looks_fine or inconclusive. A run is flagged by a Grader when its conversation’s latest verdict from the live version is flagged. The dashboard shows these flags on runs and lets you filter by them.

sikaru agent grader samples refunds

samples lists the Grader’s recent verdicts, flagged conversations first, up to 50.

Test a Grader

sikaru agent grader test NAME scores the local file on the agent’s recent runs with the file’s own judge. It saves no Grader and no verdict.

sikaru agent grader test refunds --last 20
Exit codeMeaning
0Scored within the ceiling, or the file’s release check is off.
1The ceiling would block, the test failed, or the request was refused.
2Invalid input, sign-in needed, or no recent runs to score.

sikaru agent grader run NAME re-scores recent production runs with the Grader’s live version, or its newest version when none is live. It waits and prints the flagged count. --days sets how far back to look, from 1 to 90 days (default 30).

Release checks

A release check compares a new version of the agent with the live one before the new version goes live. It needs a live version of the Grader.

When you publish, Sikaru replays a set of cases on both versions and scores both with the Grader’s live version. The cases are the conversations you labeled plus the conversations the Grader flagged in the last 30 days, up to 100.

OutcomeWhen
PassedNo regression, and the flag rate is within the ceiling.
BlockedThe new version has clearly more conversations flagged only for it, or its flag rate is above the ceiling.
Couldn’t verifyToo many cases were inconclusive, or too few were conclusive to decide.

With several checks, any block holds the release; otherwise any couldn’t-verify holds it.

sikaru agent publish -p PROJECT

publish follows the checks until they settle. A held release prints what each Grader flagged and the exact command to release it anyway, then exits 1. --no-wait returns while checks run. Over the API, a held publish answers with release_check_blocked or release_check_unverified, and the publication’s status is held.

Override a held release

If you have reviewed the flagged cases and still want the version live, publish it anyway with a reason:

sikaru agent publish -p PROJECT --publication PUBLICATION_ID --anyway "Flags are expected while the refund policy changes"

Sikaru records who released it, when and why. Publish Anyway is refused if the draft, the live version or the release check settings changed since the check ran; publish again instead.

You can also relax a check while it holds a release: turn it off or raise its ceiling. That needs a reason too, and is recorded the same way.

sikaru agent graders update --project-id PROJECT --agent-id AGENT_ID --grader-id GRADER_ID \
--json '{"releaseCheck": {"enabled": true, "ceiling": 0.2}, "reason": "New policy raises expected flags"}'

Permissions

ActionCredentials
List and read Graders, samples and cost estimatesruns:read or harness:read
Read conversation excerpts in the label queueruns:read
Test a file or re-score production runsmanaged_agents:write
Label, change the judge, escalation or release check, remove a Grader file, Publish AnywayGrader admin: a signed-in member, or a project key created with Grader admin on the dashboard

CLI and API

Inside an agent folder, the sikaru agent grader commands take the Grader’s file name and find the project and agent from the folder:

CommandDoes
sikaru agent grader init NAMEWrite a starter graders/NAME.toml.
sikaru agent grader test NAMEScore the local file on recent runs; saves nothing.
sikaru agent grader label NAMELabel conversations one at a time.
sikaru agent grader listList the agent’s Graders, built-in ones included.
sikaru agent grader get NAMEShow one Grader, its versions and label counts.
sikaru agent grader samples NAMERecent verdicts, flagged first.
sikaru agent grader run NAMERe-score recent production runs.

The API lives under /v1/projects/{project_id}/agents/{agent_id}/graders and takes the agent’s ID. The same operations are on client.graders in the SDKs and under sikaru agent graders in the CLI: list, get, update, label, label_queue, samples, run, get_run, test, get_test and estimate_cost. Publish Anyway is client.agent_documents.publish_anyway.

from sikaru_sdk import CheckChoice
client.graders.update(
project_id="PROJECT", agent_id="AGENT_ID", grader_id="GRADER_ID",
release_check=CheckChoice(enabled=True, ceiling=0.1),
)

Next steps