Back to benchmarks & research
jev-aita project preview
Benchmarks & research

jev-aita

Benchmark of TypeSafe's Jev against Sonnet 5, GPT-5 nano and local LLMs on 770 Reddit AITA verdicts: Brier scores, latency and cost

Explore repository

How it uses Jev

Jev vs. LLMs on Reddit's "Am I the Asshole?"

Highlights

  • Who is actually right. Only how well each model predicts Reddit's verdict
  • Label-only use. Every model had to give four probabilities, Jev natively
  • Full-size local models. Qwen and Gemma ran as 4-bit builds with reasoning
Language
Python 100%
Last activity
Sep 2026
Added
2026-09-24
Stars at snapshot
2
Status
Community listing
aitabenchmarkcalibrationllmllm-evaluationreddit

Source: community catalog. Details and metrics reflect the published community snapshot and are not a performance endorsement. Documentation details extracted from GitHub · the README.

Listed in Jev Library

If this is your project, the badge below links back to this page. Paste it near the top of your README.

[![Listed in Jev Library](https://jevlibrary.dev/api/badge/dchristopoulos-jev-aita)](https://jevlibrary.dev/projects/dchristopoulos-jev-aita)

Something wrong on this page, or would you rather not be listed? Tell us and it will be changed.

  • jev-eval-agent project preview
    Benchmarks & research

    jev-eval-agent

    Personal-assistant agent built on Vercel's eve with 100 mocked tools, measuring how many steps it takes when Jev picks the tool versus the LLM.

    HTML 105Details
  • typesafe-ai-benchmark project preview
    Benchmarks & research

    typesafe-ai-benchmark

    This is a LLM Gateway that mimics typesafe ai structured output. Like an imposter Jev.

    TypeScript 38Details
  • jev-on-a-laptop project preview
    Benchmarks & research

    jev-on-a-laptop

    Unofficial study: Jev-style parallel typed decisions on stock 1.5B-8B models on an Apple Silicon laptop. Benchmarks, research notes, and a Hugging Face Space demo.

    Python 23Details
  • jev-benchmarks project preview
    Benchmarks & research

    jev-benchmarks

    Probability-aware evaluation for typed decision models: calibration, selective risk, latency, and reproducible benchmarks.

    Python 17Details
  • jev-lm project preview
    Benchmarks & research

    jev-lm

    A word-level language model whose output layer is Jev: n-gram drafter, Noul chunk verification, bits-per-token eval

    TypeScript 6Details
  • decider project preview
    Benchmarks & research

    decider

    One-pass typed decisions with calibrated probabilities (System One style model), fine-tuned from Qwen3.5-2B

    Python 351Details