Testing

There are two things to test: whether a meaning means what you think, and whether the code around it does the right thing with each answer. The first needs the model; the second should not.

Grading a meaning

grade(meaning, fixtures, options) judges the meaning against labeled examples and reports how it did. expected on each fixture is what a careful person would answer.

grade accepts a condition—a string, code predicate, or meaning—not a branch, query, scale, or Choice. Test the application code around those expressions with a fake transport.

import { grade, type Fixture } from "jevlish";
import { tickets, type Ticket } from "./support.js";
import { blocked } from "./vocabulary.js";

// `expected` is what a careful person would answer.
const fixtures: Fixture<Ticket>[] = tickets.map((t) => ({
  subject: t,
  expected: ["T-1", "T-3"].includes(t.id),
  note: t.id,
}));

const report = await grade(blocked, fixtures, { seenAs: (t) => ({ subject: t.subject, body: t.body }) });

report.accuracy; // correct over decided; null when nothing was decided
report.coverage; // decided over total
report.falsePositives; // decided true, expected false
report.falseNegatives; // decided false, expected true
report.abstentions; // uncertain: a policy outcome, not a wrong answer
report.misjudged.map((m) => m.fixture.note);
report.abstained.map((a) => a.fixture.note);

The report keeps abstentions apart from wrong answers, because they are different failures with different fixes:

• accuracy is correct decisions over decided fixtures, null when nothing was decided. A low accuracy means the boundary is wrong: revise including and excluding.
• coverage is decided fixtures over all fixtures. Low coverage with high accuracy means the policy is stricter than the model’s calibration needs: consider loosening yesAbove and noBelow.
• falsePositives, falseNegatives, abstentions, and the per-fixture misjudged and abstained lists say which examples to look at. Each carries the judgment, so the probability behind a miss is right there.

options.seenAs projects fixtures the way production does; options.policy grades under different thresholds without changing the runtime.

The sample apps run this as npm run measure.

A fake transport

For everything else, give the runtime a transport that answers without a network. The jevlish/testing entry point provides a scripted fake transport and answer builders for Noul, Choice, and Score questions.

import { createSense } from "jevlish";
import { fake, yes } from "jevlish/testing";
import { ticket } from "./support.js";
import { blocked } from "./vocabulary.js";

// This transport answers every question locally and records each request.
const alwaysYes = fake(() => yes(0.95));

const sense = createSense({ client: alwaysYes });

const judgment = await sense.given(ticket).when(blocked);
judgment.status; // "decided"
alwaysYes.calls.length; // 1
alwaysYes.calls[0]?.state; // what the library would have sent

With a fake you can assert on client.calls, drive each branch of a decision with yes(0.95), yes(0.05), or yes(0.5), and test that uncertain reaches the code that handles it. pick(...) builds Choice responses and rate(...) builds Score responses.

createSense({ client }) keeps the fake out of the shared runtime, so tests do not leak into each other or into production code that uses the module-level given.