JudgeBench

JudgeBench is the rigorous benchmark for objectively stress-testing LLM-based judges on factual and logical correctness, not just human preference alignment.

This is JudgeBench: a novel evaluation framework designed to rigorously test LLM-based judges, moving beyond subjective human preferences. We utilize a specialized pipeline to convert existing challenging datasets into 350 curated response pairs, forcing the judge model to detect subtle factual or logical errors. The results are clear: JudgeBench poses a significantly greater challenge than previous benchmarks, with top-tier models like GPT-4o often performing only slightly better than random guessing and the strongest model (Claude-3.5-Sonnet) achieving just 64% accuracy. This benchmark provides the reliable platform required to assess and advance the next generation of LLM evaluators.

https://github.com/ScalerLab/JudgeBench