Evaluate traces and review results

An evaluation job applies a rubric to retained trace targets. Trace ingestion and evaluation are separate operations: an ingest receipt does not mean a judgment exists.

The flow: pick a retained trace, submit a job against its spans or messages, read the result records, and review them before you make an explicit release decision.

Submit a job

Start with a trace already available in your project. Select an actual span or message from that trace, including its account identity. The evaluator and revision identify the evaluation configuration you intend to use.

from sikaru_api import JudgmentTarget, TargetInput
job = client.evaluation_jobs.create_job(
    project,
    request_id="case-42-quality-v1",
    evaluator="case-quality",
    revision="v1",
    rubric="Pass only if the response answers the question and cites the provided evidence.",
    environment="staging",
    targets=[TargetInput(
        trace_id=trace_id,
        target=JudgmentTarget(
            kind="span", account_id="customer-42", span_id=span_id,
        ),
    )],
)["job"]

• Replace trace_id and span_id with retained identifiers.
• A job accepts 1–100 targets.
• For a message target, use kind="message" and the matching conversation and message identifiers.
• The service freezes the target context when it admits the job, so the evaluation uses that submitted boundary.

Observe or cancel

current = client.evaluation_jobs.get_job(project, job["id"])["job"]
results = client.evaluation_results.list_results(
    project, evaluator="case-quality", environment="staging", limit=50,
)

Poll with a bounded application deadline until the job reaches its terminal status. Inspect the job and result records; do not treat acceptance as success.

Result and job listings use cursors. Keep the endpoint’s returned continuation cursor and filters when you fetch more pages. evaluation_criteria.list_criteria lists available criteria. Result records keep evaluator, revision, verdict, and public reason.

Use judgments in release review

Review failures and the supporting evidence before you approve a release. Keep evaluator revisions and targets consistent when you compare changes.

Applications that already produce their own judgments can submit them with evaluation_results.record_result. This records supplied evidence; it does not launch a managed evaluator.

CLI operations are available under evaluation-jobs, evaluation-results, and evaluation-criteria.

Next steps