Your Code Review Agent Needs Eval

Overview

You can set up a PR review agent in an afternoon with a GitHub Action or a Claude Code prompt. The hard part is knowing whether you can trust it. That's what an eval is for. An eval is a set of examples where you already know the right answer, so you can score a PR agent against those answers and track how it performs over time. If you're new to evals, I covered the basics in Evaluating LLM Systems.

In this post, we'll go over why PR review evals can be challenging to build, walk through an architecture for running them, and show how to build the eval from your own PR history. We'll use Harbor to handle orchestration.

Code Sample

The sample-code-eval-harvester repo contains a CLI that builds PR review evals from your own repository. This post goes into detail on how to approach PR agent evals, but in 2026 nobody expects you to do this by hand. Point a coding agent like Claude Code or Codex at the CLI and let it do the work: finding PRs to harvest, labeling review comments, and building and verifying the Harbor tasks. It's still important to understand how it works, so you can verify the eval represents your codebase.

Problems

There are two problems to solve. (1) Generic benchmarks aren't representative of your code base, so the examples have to come from your own code which requires effort to extract. (2) Every task needs its own environment. Think about how long it takes a new dev to get your code running locally, then repeat that for every task. That same environment also has to seal the answer away, because agents can't be trusted not to look.

Problem 1: You Need Examples From Your Own Code

Take a public benchmark like Code Review Bench. It scores review agents on 50 PRs from five open-source projects (Sentry, Grafana, Cal.com, Discourse, and Keycloak), each with human-verified comments. Those comments flag real bugs, like a race condition in Grafana's anonymous device limit or a Sentry endpoint that crashes when an API token has no org member. Catching those shows the agent can find generic bugs. It doesn't show whether the agent knows the rules only your team enforces.

Let's walk through an example. A coding agent opens a PR that moves plan IDs out of Stripe metadata and into your database, not knowing that a separate billing-webhooks service reads that metadata every time a subscription renews:

 export async function changePlan(sub: Subscription, newPlan: Plan) {
   await stripe.subscriptions.update(sub.stripeId, {
     items: [{ id: sub.stripeItemId, price: newPlan.stripePriceId }],
     proration_behavior: "none",
-    metadata: { planId: newPlan.id },
   });
+  await db.subscriptions.update(sub.id, { planId: newPlan.id });
   await entitlements.set(sub.customerId, newPlan.id);
 }

Here's what two inline review comments on this PR might look like.

A generic PR agent sees a reasonable cleanup and approves:

change-plan.ts:6 Nice cleanup. Consider retrying this write if it fails so Stripe and the database can't drift apart.

A reviewer who knows your system, whether that's a human or an agent with the right context, requests changes:

change-plan.ts:5 billing-webhooks reads planId from Stripe metadata on every renewal. Removing it here resets anyone who changes plans back to their old plan at their next renewal, up to a month from now.

The bug sits in the seam between two services. Each service's tests pass on their own, and nothing fails until renewals run a month later. Public benchmarks don't contain bugs like this because they don't know how your services fit together. A PR like this one, with the second comment as the answer key, is exactly the kind of datapoint your eval needs.

Problem 2: Every Task Needs a Sealed Environment

Each task needs the repository at the right commit, with everything the agent needs to build, run, and explore the code. Doing that by hand for dozens of tasks doesn't scale.

The environment also has to hide the answer. The fix is in a later commit, and the reviewer's comments are on GitHub. If the agent can reach either one, it can look up the answer instead of reviewing the code. The scorer still needs that answer to grade the review, so it has to live somewhere the agent can't reach.

Approach

Before getting into how to build the dataset, let's look at the approach. This is generally how agent evals get run at scale. Each example becomes a self-contained task, and a runner fans those tasks out to isolated sandboxes on a remote compute layer instead of running them on your laptop. Inside each sandbox, the review agent can only reach the hosts it needs, like its LLM API.

Step 1: Each PR from your history becomes one task. A task has an instruction for the agent, an environment with the repository rewound to before the change with the change applied on top, and a tests/ folder with the answer key and the scoring code.

Step 2: Harbor reads the tasks and schedules trials. A trial is one agent reviewing one task. You can run the same tasks against several agent configurations at once, like two prompts or two models, to compare them.

Step 3: Each trial runs in its own sandbox on remote compute. We use microVMs, but the substrate can change. Lambda MicroVMs, Kata Containers on Kubernetes, or anything else that isolates trials from each other and restricts the network works. Running remotely is what lets you fan out dozens of trials at once instead of queuing them on one machine. The network is restricted to an allowlist: the agent's LLM API, plus any package registries the environment needs to build. GitHub isn't on it, so the agent can't look up the fix or the review thread.

Step 4: The agent writes its findings, and Harbor copies them to a verifier that runs in a separate environment. The answer key only ever lives on the verifier side, so the agent can't read it or tamper with it. The verifier compares the findings to the answer key and writes a score.

Step 5: Scores roll up into a scoreboard. You can slice it by agent configuration to pick the better prompt, and by risk level to decide whether the agent is good enough to auto-approve low-risk PRs.

The rest of this post covers what goes into a task, how to build tasks from your PR history, how scoring works, and how to seal the environment.

What's in a Task

Harbor packages agent evals as self-contained task directories. Because the format is standard, you can swap the agent, model, or prompt without touching the dataset, and anyone can rerun the eval and get the same result.

Each datapoint is one task directory:

tasks/pr-13611/
├── task.toml              # agent
├── instruction.md         # agent
├── environment/
│   └── Dockerfile         # agent: rewinds and seals the repo
├── tests/                 # verifier only
│   ├── Dockerfile
│   ├── test.sh
│   ├── score.py
│   ├── oracle.json
│   ├── rubric.md
│   └── judge.toml
└── solution/
    └── solve.sh           # reference submission

The important split is between what the agent sees and what only the verifier sees. The agent gets the instruction and an environment with the repository. Everything under tests/ belongs to the verifier and is never mounted into the agent's environment.

instruction.md is what the agent reads, and it's the same for every task. It asks for defects and a verdict and never mentions the PR, the reviewer, or what they found. If the instruction hints at the answer, you're measuring reading comprehension instead of review.

# Review this change
 
You are reviewing one change to this repository, presented as `change.patch`
applied over the repository at its base state. Read the change and review it as a
competent reviewer would.
 
Report the defects a reviewer should catch. For each defect, give the file and
line, a one-sentence statement of the problem, and a severity (`low`, `medium`,
or `high`). Then give an overall verdict: approve or request changes.

task.toml holds timeouts, network access, and verifier settings. The two settings that matter most are network_mode = "allowlist" for the agent and environment_mode = "separate" for the verifier. The allowlist holds your LLM API and the package registries your build needs, and never your source forge. Harbor defaults network_mode to public, so make sure you set it.

[agent]
timeout_sec = 1800.0
 
[environment]
network_mode = "allowlist"
allowed_hosts = [
    "llm-api.example.com",  # your LLM API
    "pypi.org",
    "*.pythonhosted.org",
    "registry.npmjs.org",
    "crates.io",
    # ...whatever else the build needs, but never github.com
]
os = "linux"
 
[verifier]
timeout_sec = 1800.0
environment_mode = "separate"
 
[metadata.harvest]
kind = "reject"
expected_verdict = "block"
change_risk_structural = "medium"
change_risk_classified = "high"
rubric_version = "v1"

environment/Dockerfile builds the repository at the base commit with the change applied. We'll cover it in the sealing section below.

tests/oracle.json is the answer key. It holds the expected verdict and the reviewer's findings, each with a severity and location. Here's the one for the plan change example:

{
  "expected_verdict": "block",
  "blocking_severity": "high",
  "findings": [
    {
      "id": 1,
      "statement": "Removing planId from Stripe metadata breaks billing-webhooks, which reads it on invoice.paid, so customers who change plans are reset to their old plan at the next renewal.",
      "severity": "high",
      "reference_only": false,
      "blocking": true,
      "locations": [{ "path": "services/subscriptions/src/change-plan.ts", "line_start": 2, "line_end": 7 }]
    }
  ]
}

tests/score.py and tests/judge.toml hold the scoring logic and the pinned judge config, which we'll cover in the scoring section. The scorer writes a reward file that keeps each dimension separate, so you can see why an agent scored the way it did:

{
  "coverage_required": 1.0,
  "coverage_all": 0.5,
  "precision_strict": 0.33,
  "credited_high": 1,
  "uncredited": 2
}

tests/test.sh is the verifier's entry point. It clears any environment variables that could override the pinned judge, runs the scorer, and always writes a reward file, even if scoring crashes. A failed run should score zero instead of quietly disappearing from your results.

solution/solve.sh submits the answer key as if it were the agent's output. It's how you sanity check the scorer.

Building Tasks From Your PR History

Now that we know what a task looks like, let's talk about where they come from. The good news is you already have a labeled dataset. Every PR a reviewer blocked records the change they rejected, what they said was wrong, and the fix they approved. The work is turning that history into tasks.

Find Datapoints

A PR that went through a round of "request changes" gives you two datapoints:

• Reject: the version the reviewer rejected. The expected verdict is "block," and the reviewer's comments are the defects the agent should find.
• Approve: the version that was approved. The expected verdict is "approve." These matter as much as the rejects, because an agent that blocks everything is useless.

Getting the base commit right is critical. The agent has to see the code before the change, with the change applied on top. If you base it after the fix, the bug is already gone.

Not every PR is usable. PRs that merged without a review have no verdict to grade against, and PRs whose reviewed commits were force-pushed away can't be rebuilt. If most of your PRs merge without review, you'll find out quickly.

Try to keep a balanced mix. You want enough real defects to test whether the agent catches them, enough clean changes to test whether it approves good work, and enough low-risk changes to decide whether you can automate their approval.

Label What the Reviewers Found

Every review comment gets one of five labels:

• Defect: a problem the reviewer would block on.
• Nit: a style or preference comment that doesn't block.
• Question: a request for information.
• Approval: a sign-off with no finding.
• Bot: an automated comment from a linter, coverage tool, or dependency bot.

The most important call is defect vs. nit. If you label a nit as a defect, the eval rewards an agent for nitpicking, and you'll end up tuning it to be noisier. A PR with only nits is an approve datapoint.

Each defect also gets a severity of low, medium, or high, and only high blocks a merge. Every severity needs evidence, like a file and line, an error, or an output, along with the rubric rule that assigned it. Without that, a severity is just an opinion, and a dataset of opinions gives you a score you can't defend.

Build the rubric from your repo's conventions. Start with a standard review baseline and add what your CONTRIBUTING.md, CODEOWNERS, and style guides say. Version it alongside the dataset so you know which rules each score was graded against.

Here's how a few comments on the plan change PR would be labeled:

Classify Risk

Severity describes what a reviewer found. Risk describes how dangerous the change is in the first place. These are separate. A low-risk change can hide a high-severity bug, and a high-risk change can be perfectly clean. If you mix them up, the dataset can't answer either question.

Risk is what decides whether the agent is allowed to auto-approve. Most teams don't want an agent approving everything. They want it approving low-risk changes and sending the rest to a human. To trust that, you need the agent's score on the low-risk slice by itself, which an overall score won't tell you.

There are two ways to classify risk, and we use both:

• Structural: a file that maps path prefixes to risk levels. Most teams already keep risky code in known places like auth/, billing/, and migrations/, so the directory layout does most of the work. It's cheap and easy to audit.
• Judgment: a model or person classifies what the layout misses, like a one-line config change that flips a production feature flag.

Record both and flag where they disagree. When a change touches several paths, the highest risk wins. Unmapped paths default to medium, never low, because under-classifying a risky change is the expensive mistake.

Then check the distribution. If only a handful of datapoints are low risk, you don't have enough data to automate approvals for that class yet. The grid below shows why the low-risk, high-severity cell is the one to watch.

Scoring

Senior SWE-Bench has a great way of scoring agents. It grades a coding agent on several dimensions (behavioral tests, task rubrics, and a taste judge) and rolls them into one metric. We take a modified approach for PR review. Senior SWE-Bench grades the code an agent writes, while here the patch is the input and the review is what gets graded.

For each task, the agent returns findings (file, line, statement, severity) and a verdict. From those, the scorer computes a mix of numerical and categorical inputs:

• Required coverage: the fraction of blocking findings it caught.
• Overall coverage: the fraction of all findings it caught.
• Finding precision: the fraction of its findings that matched a real finding.
• Severity agreement: whether it rated findings the same way the reviewer did.
• Verdict: approve or request changes.

Those inputs feed one decision: did the agent get this PR right? You can make that call with a heuristic, like "pass only if the verdict matches and required coverage is 1.0," or hand the inputs to an LLM judge with a rubric and let it decide. A heuristic is cheap and predictable. A judge handles the gray areas, like a finding that's right but loosely described.

The output is categorical: pass or fail on each task, along with the block or approve call behind it. That's what makes results easy to summarize across the dataset. You can report:

• Accuracy: the fraction of tasks the agent got right.
• Block recall: of the PRs that should have been blocked, how many it blocked.
• Block precision: of the PRs it blocked, how many deserved it.
• False approve rate: how often it let a bad PR through. This is the expensive mistake.
• False block rate: how often it blocked a clean PR. This is the noise your engineers feel.

Slice any of these by risk level to see whether the agent is safe to auto-approve low-risk PRs.

The inputs are what tell you why a decision went the way it did. When a task fails, low required coverage means the agent missed the blocking bug, low finding precision means it buried the review in noise, and low severity agreement means it found the right problem but didn't think it mattered.

Matching findings happens in two stages. First, narrow the candidates by file and line. Then an LLM judge decides whether two findings in the same spot describe the same problem. Pin the judge's model and set temperature to 0. If the judge changes between runs, your scores aren't comparable.

When the agent flags something the reviewer didn't, count it as uncredited instead of wrong. Reviewers miss things, so the review you mined is an incomplete answer key. Penalizing the agent for finding a real bug the reviewer missed teaches it to find less.

Test the scorer before you trust it. Run solve.sh to submit the answer key as the agent's output. Anything short of a perfect score means the scorer is broken.

Sealing and Running

Rewind the Repo

The agent needs the repository as the reviewer saw it. The Dockerfile shallow clones the repo, fetches only the base commit (the parent of the change), and resets to it. It then applies the change as uncommitted edits. The agent sees one commit of history with a diff on top, which is the same thing a reviewer sees when they open the PR.

Seal Off the Answer

If the agent can find the fix or the review thread, you're measuring search instead of review. We seal at three layers.

Git. A shallow clone isn't enough on its own, because git keeps old state in places that are easy to miss, like remote-tracking refs and the reflog. The Dockerfile removes the remote, deletes remote-tracking refs, expires the reflog, and garbage collects anything unreachable:

FROM public.ecr.aws/docker/library/buildpack-deps:bookworm-scm
WORKDIR /work
 
# Only the base commit's tree, with no history or tags around it.
RUN git clone --depth 1 --single-branch --no-tags "$REPO_URL" repo
WORKDIR /work/repo
RUN git fetch --depth 1 origin $BASE_COMMIT \
 && git reset --hard $BASE_COMMIT
 
# Seal: no remote, no remote refs, no reflog, nothing unreachable.
RUN git remote remove origin \
 && git for-each-ref --format='%(refname)' refs/remotes | xargs -r -n1 git update-ref -d \
 && git reflog expire --expire=now --all \
 && git gc --prune=now --aggressive
 
# Apply the change under review as uncommitted edits.
COPY change.patch /work/change.patch
RUN git apply --whitespace=nowarn /work/change.patch \
 && rm -f /work/change.patch

Network. The agent can only reach hosts on the allowlist, like its LLM API and package registries. GitHub isn't on it, so the agent can't look up the PR.

Verifier. Scoring runs in a separate environment. The answer key never enters the agent's environment, so the agent can't read it or tamper with it.

Verify the seal before you use a task. Scan everything the agent can see (the instruction, the patch, and the repository) for the PR number, the review text, and the verdict. Plant a known token first and confirm the scan finds it. Otherwise a scan that silently didn't run looks the same as a clean one.

Run in Parallel

Tasks are independent, so running the eval is really a container orchestration problem. You fan out across every task, often across several agent configurations at once, and collect the rewards into one scoreboard.

MicroVMs work well here. Each trial is isolated from the others, the network policy is fixed at launch so the agent can't loosen it, and the image is built once and reused for every trial. One catch: some microVM platforms only offer full internet egress or none. On those, route egress through a VPC with a firewall that only permits the allowlisted hosts.

Conclusion

A review agent is easy to set up and hard to trust, and generic benchmarks can't tell you how it performs on your code. Your PR history can. Harvest the PRs your reviewers blocked, separate real defects from nits, classify risk separately from severity, and score the agent on one decision: would it let this PR through? Package each example as a sealed Harbor task and run them in parallel.

Start small. Ten to twenty PRs is enough to compare two prompts and see where your agent misses. Before turning on auto-approval, look at the low-risk slice on its own and make sure it has enough datapoints to trust.