Evidence and tracing

Results include the requests that left the process, what came back, how each answer was read, and the policy that read it. You can inspect that evidence without repeating the model call.

Judgment

Judgment<T> is the fundamental return type:

type Judgment<T> =
  | { status: "decided"; value: T; evidence: Evidence }
  | { status: "uncertain"; evidence: Evidence };

“Decided” means the answer passed the acceptance policy, not that it is a proven fact. isDecided(judgment) is a type guard; truthOfJudgment(judgment) collapses a boolean judgment back to true | false | "uncertain".

Evidence

import { given } from "jevlish";
import { ticket } from "./support.js";
import { needsAttention } from "./vocabulary.js";

const judgment = await given(ticket)
  .seenAs((t) => ({ subject: t.subject, body: t.body }))
  .when(needsAttention);

const { requests, judgments, policy } = judgment.evidence;

for (const record of judgments) {
  switch (record.kind) {
    case "code":
      record.truth; // boolean; the predicate's name is record.label
      break;
    case "noul":
      record.probability; // P(yes)
      record.truth; // true | false | "uncertain", after the policy
      break;
    case "choice":
      record.choice; // the option id; record.probabilities has the distribution
      break;
    case "score":
      record.score; // the expected position on the scale
      break;
  }
}

for (const request of requests) {
  request.state; // what the model saw
  request.questions; // what it was asked
  request.usage.input_tokens; // what it cost
  request.cached; // whether a cache answered instead
}

policy.noul.yesAbove; // the thresholds that turned probabilities into truth

evidence.judgments records every judgment that contributed, in one of four shapes:

• code: a predicate that ran locally, with its label and boolean truth. Recorded alongside model answers so the trace shows the whole decision, not only the part that involved inference.
• noul: a proposition, with the instructions sent, the probability of yes, and the truth the policy assigned.
• choice: a Choice, with the chosen option, confidence, the full probabilities, and whether it was accepted.
• score: a measurement, with the score, confidence, probabilities, and accepted.

evidence.requests records each request: the state the model saw, the questions, the answers, token usage, and whether the response was cached. Summing usage.input_tokens over a query’s requests is what it cost.

evidence.policy is the policy in force, after any withPolicy override.

Where evidence comes from

• A predicate or branch: judgment.evidence, or decision.judgment.evidence.
• A query: result.evidence, covering every item.
• ask: every answer shares one evidence object.
• grade: each misjudged and abstained entry carries its fixture’s judgment, so a wrong answer can be traced to its probability.

The sample apps’ npm run trace commands are nothing more than printing this structure. See the Support desk example.