EvalArena: A Simple Way to Find Out If Your Fine-Tune Actually Helped

Written by

in

If you have ever fine-tuned a model, you already know the uncomfortable question that comes right after: did this actually make things better, or does it just feel different? Automated metrics only get you so far. For open-ended output quality, the only judgment that really counts is a human reading two answers side by side and picking the one they would rather receive. That is the entire idea behind EvalArena.

EvalArena is a lightweight comparison tool built for exactly one job: take a set of prompts, pair each one with two candidate outputs, and let human reviewers vote on which is better. The votes roll up into a win rate with a proper confidence interval, broken down overall and by category, so you can see not just whether one model wins, but where it wins and where it still falls behind.

Why blind voting matters

A comparison tool is only as trustworthy as its blinding. If a reviewer can tell which answer came from the new fine-tune, their vote is contaminated before they even start reading. EvalArena strips away every label before a prompt reaches a reviewer, and it randomizes which side each answer appears on for every single reviewer. That second part matters more than people expect. Reviewers reliably favor whichever answer they see first, regardless of quality, so without position randomization your results are measuring a seating chart, not a model.

What you get out of it

Once votes are in, EvalArena computes a win rate with a Wilson score confidence interval, which is the right way to express uncertainty on small sample sizes rather than pretending a percentage from twelve prompts is a precise number. Results break down by category too, so if you tagged your prompts as reasoning, tone, or factual accuracy, you can see exactly where the fine-tune helped and where it did not.

For teams comparing more than two candidates at once, there is a tournament mode. Instead of a single win rate, it computes an Elo-style ranking across every model in the arena, the same style of rating used to rank chess players, applied here to rank language model outputs against each other.

There is also an optional auto-pilot mode, where a judge model votes first on every prompt. Clear-cut cases get resolved automatically, and only the close calls or disagreements get routed to a human reviewer. For a large prompt set, this can cut the amount of human review time dramatically while still keeping a person in the loop for the cases that actually need judgment.

When reviews are done, you can export a clean, shareable HTML report summarizing the whole arena. It is meant to be dropped straight into a Slack message or a blog post making the case for or against a rollout.

How it is built

EvalArena has three parts that all talk to the same underlying engine: a command-line tool for scripting arena creation as part of a CI pipeline, a small web server with a browser-based reviewer interface for teams who want to vote without installing anything, and a results dashboard for digging into the numbers afterward. The reviewer interface supports keyboard shortcuts, so a reviewer working through fifty prompts can fly through them with the arrow keys and the space bar instead of hunting for a mouse.

Who this is for

If you run fine-tuning jobs and want a repeatable way to check whether a new version is actually an improvement, if you are A/B testing two versions of a prompt and want more than a gut feeling, or if you just want a fair, blinded way to compare a handful of models on the same task, EvalArena is built for that exact moment. It is a small, focused tool that answers one question well: which one is actually better?

Let’s explore it on GitHub: https://github.com/AIHelpers/evalarena

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *