Benchmarks
Measured on real work.
We're building evaluations grounded in the messy, ambiguous work models are actually asked to do.
Public benchmark · Yet to be released
What it will measure
Built to stay meaningful.
Our public benchmark is in final validation. When it launches, every result will ship with its full methodology.
01
Real-world tasks
Built from production pull requests and review threads, not puzzles written for a test.
02
Multi-turn by default
Tasks unfold over several turns with clarifications, changing constraints, and follow-ups.
03
Expert-scored
Every answer is graded by domain experts against calibrated rubrics, then blind re-reviewed.
04
Held out and refreshed
The scoring set is never published and is refreshed regularly to stay ahead of contamination.
Available now
Need an evaluation before the public launch?
We build private, held-out benchmark suites for partners today, scoped to the capabilities your model needs to prove.