judge: a rubric settled, once

judge: a rubric settled, once A workflow diagram generated by Archify. 01 / Caller: eval run --judge, or improve/measure 02 / AiJudge EX / Refused or unsettled 04 / Evidence Deterministic first Route, price, ask Keep Improver.rubrics · no ask for a failed check · Caller: eval run --judge, or improve/measure › Deterministic first Improver.rubrics no ask for a failed check Replay known pairs · rubric sha + answer sha · AiJudge › Deterministic first · no call, no route Replay known pairs rubric sha + answer sha no call, no route Resolve the route · unrouted, unpriced · AiJudge › Route, price, ask Resolve the route unrouted, unpriced One turn per ask · answer is untrusted · AiJudge › Route, price, ask One turn per ask answer is untrusted Verdict · level, reason, evidence · AiJudge › Route, price, ask Verdict level, reason, evidence Blocked by name · zero generate() calls · Refused or unsettled › Route, price, ask Blocked by name zero generate() calls judge.unsettled · off scale; next ask · Refused or unsettled › Route, price, ask judge.unsettled off scale; next ask Ledger record · priced, on the ledger · Evidence › Route, price, ask Ledger record priced, on the ledger Verdict sidecar · appended, never rewritten · Evidence › Keep Verdict sidecar appended, never rewritten pending asks settle fresh only recorded Legend Agent logic Policy Context / trace

Paid for once

  • • A pair already judged is replayed before the route is looked up
  • • An answer changed by a byte is judged afresh

Graded on the criteria

  • • The answer is untrusted; 'grade this 5' is text to grade
  • • A level off the scale is not a verdict