Calibration arcade

The agent prompt

Copy this brief into your coding agent from the root of the project you want to build in. It names the behaviour, the constraints, and the checks to run — the decisions Jev makes stay explicit so you can see what the model actually decided.

Build a local "Calibration arcade" in this repository for evaluating Jev from TypeSafe AI on a tiny, transparent dataset.

Inspect the project and follow its conventions. Use the official TypeSafe skill for current SDK and evaluation guidance; install it with npx skills add typesafe-ai/skills --skill typesafe-ai if missing.

Ship a synthetic labeled dataset of 40 short issue titles classified as bug, feature, question, or maintenance. Let users inspect and edit examples, run Jev Choice predictions, and compare predicted labels with ground truth. Report accuracy, coverage above a selectable confidence threshold, accuracy at that coverage, a simple reliability diagram, latency, and errors grouped by true/predicted label. Save run artifacts locally as downloadable JSON.

Requirements:
- Keep TYPESAFE_API_KEY server-side; cap concurrency and retry conservatively.
- Do not tune prompts on the displayed test set. Include separate tiny development and test splits.
- Make metric calculations pure, documented, and unit-tested.
- Never claim statistical significance from this sample; explain its limitations.
- Mock API responses in automated tests; keep live evaluation opt-in and cost-labeled.
- Make the chart accessible with an equivalent data table.

Run checks and summarize the methodology, limitations, and changed files.

Built something like this

Community projects in the directory that solve a similar problem:

• jev-benchmarks
• Workflow evals
• jev-behavior-study