What is CyberGym?

Most AI security benchmarks ask a model a question. CyberGym hands the model a broken program and asks it to prove the bug is real: produce an input that crashes the unpatched build and leaves the patched one untouched. The UC Berkeley benchmark covers 1,507 real vulnerabilities from 188 widely used open-source projects, every one of them originally found by Google's OSS-Fuzz fuzzing campaign, according to the project. What follows: what the benchmark measures, how its scoring works, how to run it yourself, and how to read a leaderboard row without fooling yourself.

Where the tasks come from

OSS-Fuzz is Google's continuous fuzzing service, launched in 2016 after the Heartbleed vulnerability in OpenSSL, per its own documentation. That history is what makes CyberGym's raw material different from a synthetic exercise set: every instance is a real vulnerability a fuzzer found and a maintainer patched, and the task ships the repository checked out just before the fix.

The authors — Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang and Dawn Song — frame the benchmark as a response to security evaluations that were, in their words, "based on small-scale benchmarks and only measure static outcomes" (the paper, first posted to arXiv in June 2025 and revised in March 2026, is listed by the authors as an ICLR 2026 publication). CyberGym's answer is scale plus ground truth: repositories spanning thousands of files and millions of lines, and a patched build that acts as an objective referee.

The Level 1 task: write a PoC that works

The headline task — Level 1 on the leaderboard — gives the agent a text description of the vulnerability and the unpatched codebase, and asks it to generate a proof-of-concept input that reproduces the bug. Success is verified differentially: the submission server runs the PoC against the vulnerable build, where it must crash, and against the patched build, where it must not. The agent itself never sees the fix — only the verification server touches the patched image, the project's FAQ states.

The benchmark actually comes as a difficulty ladder defined by how much information you hand over. At Level 0, with no vulnerability description, the project reports only 3.5% of instances get reproduced. Level 2 adds the crash's stack trace; Level 3 hands over the ground-truth patch, and both greatly raise reproduction rates relative to Level 1.

Two findings from the project's own evaluation to internalize before you look at any score. First, longer exploits are much harder: agents succeed only about 10% of the time on instances whose reference PoC runs over 100 bytes, and those make up 65.7% of the benchmark. Second, reasoning effort matters — on a 300-task subset, raising GPT-5's reasoning level moved its success rate from 7.7% to 22.0%.

How the scoring works

Success rate is the percentage of instances where the agent reproduced the vulnerability with a working PoC. Three mechanics underneath decide what a number actually means.

One submission, not a pile. An agent can submit several PoCs while working, and the FAQ describes two ways to score: count the task solved if any submitted PoC works, or only if the PoC the agent designates as its final answer works. CyberGym requires the final-submission metric, because as capability improves the gap widens and the any-of rule "increasingly rewards brute-forcing."

Trials are not free. The leaderboard legend says an instance counts as solved if any trial succeeds, and rows declare how many attempts they used. In the leaderboard's published data, 78 of the 82 Level 1 rows report a single attempt per task; the four rows sourced from Anthropic's Claude Sonnet 4.5 system card report 30. A score earned over 30 swings is not directly comparable to one earned over a single swing.

Teams run and submit it themselves. The site says it plainly: "Results are evaluated and submitted by individual teams; agent runs are stochastic, so scores may vary across evaluations." The project's submission guidelines, dated August 2026, back that with requirements — a structured report, a public writeup of the full experimental setup, example trajectories and logs for at least ten tasks, exit codes for every instance, and, newer, cost alongside success: "The most capable models under unconstrained resources have nearly saturated the benchmark," the guidelines say, so future submissions must report token and dollar estimates per task.

Two row labels change the setup rather than the difficulty: dynamic means the agent ran against a sanitized vulnerable container or compiled binary; test-time mem. means it built a knowledge base across instances. Both are permitted, and both must be disclosed.

Running it yourself

CyberGym is open source under Apache 2.0 — a Python package, a Docker-based harness and example agents in the project repository, with the dataset on Hugging Face. The catch is weight. The benchmark data is about 240GB; the full set of compilation environments is roughly 10TB; a binary-only mode cuts that to about 130GB if you only need static analysis; and the repo offers a ten-task subset (five tasks agents tend to solve, five they tend not to) for a first run.

The loop is: start a local PoC submission server, generate a task — you get the vulnerability description, the unpatched repository archive and a submission helper — run your agent against it, then verify the PoCs the agent filed. A bundled firewall proxy can confine agent containers to an allowlist of domains, which matters because the FAQ is blunt: network access is not needed to solve tasks and opens the door to reward hacking — an agent with the open internet can simply look up the fix. Runs that enable it, or a runnable environment, must be disclosed, with trajectories audited for shortcuts.

One rule the README puts in bold: deploy everything locally, never expose any part of CyberGym to the public internet — the submission endpoint executes uploaded inputs against real unpatched vulnerabilities, and the shipped API key is a placeholder.

How to read a leaderboard number honestly

The Level 1 leaderboard carried 82 rows when we checked: 49 tagged model-focused and 33 agent-focused, the two categories defined in the submission schema. Model-focused means the run was designed to test the model's underlying capability rather than a specialized agent design. In practice even model-lane rows name the harness that drove the model — Claude Code, Google Antigravity, OpenAI Agent, XekRung Agent — so a score is a model-plus-scaffold number, not a bare model number.

The top rows at the time of writing are 88.9% for Alibaba Security's XekRung-1.5-27B-Preview in the model lane and 99.2% for the multi-model Lyrie agent in the agent lane. We covered the first result in our leaderboard brief, including its self-reported provenance. Four rules follow from the mechanics above:

Differences inside a few points are noise. The site warns it directly: "Vulnerability descriptions can be ambiguous, and with leading systems already achieving high performance, modest score differences may not reflect meaningful capability gaps." The top ten model rows span 88.9% to 80.0% — most of that band separates harness choices, not capabilities.
Check the trial count before comparing. One attempt and thirty attempts produce different-shaped numbers under an any-trial-succeeds rule.
Check the disclosures. Network access, dynamic environments, test-time memory and harness all move scores; every one of them is supposed to be in the writeup, and none of them are independently re-run by the project.
No row has been independently reproduced. Every number on the board is produced by the team that benefits from it. That does not make it wrong; it makes it a claim with artifacts attached, not a measurement you have inherited.

For calibration, the paper's own evaluation found even its best agent-model combinations solving only about 20% of tasks — a reminder that today's 90%-plus rows reflect scaffolds, disclosure settings and time passed, not a fixed property of the benchmark.

How it differs from other security benchmarks

The nearest neighbors measure something else. Cybench, EnIGMA and NYU CTF Bench are Capture the Flag benchmarks — Cybench alone packages 40 professional-level CTF tasks from four competitions — where the agent cracks a contest puzzle for a flag. CyberGym tasks are production bugs from real projects with a maintainer's fix as ground truth, and scoring is an executable differential test rather than a string match. Meta's CyberSecEval sits on the other side of the discipline: it measures propensity — whether a model writes insecure code, and whether it complies when asked to help with an attack — not whether it can find and trigger a flaw in a real codebase.

For a practitioner: CTF scores say how well an agent plays security puzzles, safety-suite scores say how risky a model is as an assistant, and a CyberGym score says how often an agent can take a described vulnerability in a real project and prove it with a working exploit.

What the agents found beyond the leaderboard

The project also reports real-world output beyond the scored task: PoCs that crashed patched builds led to manual confirmation of incomplete patches and, after deduplication, 10 unique previously unknown zero-days, each persisting an average of 969 days before discovery. In open-ended mode — agents pointed at the latest codebases across 431 OSS-Fuzz projects with no prior knowledge of existing bugs — the team reports GPT-5 triggered 56 crashes yielding 22 confirmed zero-days, against 7 for GPT-4.1. The site's headline totals: 34 zero-days and 18 historically incomplete patches. Those are the project's own numbers, validated by the team itself; treat them with the same provenance caveat as the leaderboard.

What we could not verify

Venue details. The ICLR 2026 status comes from the project's citation block; OpenReview blocked our automated access, so we could not confirm it on the venue page itself.
The zero-day arithmetic. The site's summary tiles say 34 zero-days and 18 incomplete patches, while its detail sections describe 10 unique zero-days from post-patch crash validation, 7 and 22 from open-ended scans (4 overlapping), and 17 confirmed incomplete-patch cases across 15 projects. The sub-figures do not transparently sum to the totals; we could not reconcile them.
Any leaderboard row, independently. We did not re-run the benchmark, and found no third-party re-run of any submission. Cost figures in submissions are explicitly estimates.
Terminology. "Model track" and "agent track" are shorthand — the schema's own field values are model and agent categories.