GPT-6.1 Sol migrates COBOL at 97.9% — GPT-5.6 scored 5.4%
OpenAI published two evaluations on Monday night that nobody had on the calendar, and they land on the most expensive legacy problem in enterprise software.
OpenAI says GPT-6.1 Sol migrates COBOL programs to C++, Java, Kotlin, Python or Rust with 86.6% accuracy at low reasoning effort — against 5.4% for GPT-5.6 Sol at the same setting and at 61% of the API cost — rising to 97.9% at higher reasoning effort, which the company concedes "nearly saturates" its own test. The numbers come from a new internal evaluation OpenAI described in a post published Monday, alongside a second benchmark that it released as code. A 5.4% to 86.6% jump on the same task, one model generation apart, is the largest single-step improvement on a coding benchmark this year that did not come with an accompanying capability announcement.
The evaluation itself is the part worth understanding, because the scoring rule is unusually strict. Models are asked to migrate COBOL programs into one of five modern languages while preserving behavior, and they are graded on hidden tests that require byte-for-byte parity on observable outputs. A migration counts only if every test passes. There is no partial credit for a rewrite that compiles and runs.
That rule exists because of a specific COBOL trap. COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested, so a modern rewrite that rounds those values will execute cleanly and produce different financial results. Nothing throws. Nothing logs. A payroll system silently pays out fractions of a cent it should have dropped, and finances reconcile wrong at the end of the quarter. Byte-for-byte output matching is the only test that catches it, and it is why the low-reasoning score of 86.6% is a more honest number than it first appears: the model gets roughly six of every seven programs exactly right, and the seventh fails a test rather than shipping.
Why the language matters at all: COBOL still supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, per IBM figures OpenAI cites in the post, while Fortran runs weather forecasting and nuclear reactor safety analysis. These are the systems where the original developers have retired, the documentation is incomplete, and the business rules exist only in the code. OpenAI says the work was guided by enterprise customers telling it that coding models struggle with exactly these languages — which is also the reason the market for rewriting them has been billed in person-years rather than tokens.
The second benchmark shows where the improvement actually lives. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in a specified language, then checks the resulting database files against the reference SQLite implementation's test suite. It is a long-horizon test — the model has to organize a codebase, manage state, handle errors and keep old functionality working while adding new features — and the same test applies regardless of language.
On it, GPT-6 Astra averages an 87.4% test pass rate across 20 languages, up from 26.8% for GPT-5.6 Sol, with gains in every language tested. COBOL improves from 10.9% to 72.1%, OCaml from 25.6% to 85.8%, Fortran from 21.4% to 83.5%, Visual Basic from 27.5% to 89.4%. Astra clears 83% in 19 of the 20 languages. The exception is COBOL at 72% — still the hardest language in the set by a wide margin, which is a useful corrective to the migration headline: the model is now good at COBOL, not solved on it.
The cheap tier is where the story gets less comfortable. GPT-6 Luna also beats GPT-5.6 Sol in all 20 languages, averaging 40.3% at an estimated $0.77 per sample against $3.48 for the older model — roughly 78% cheaper. But on COBOL specifically, Luna moves from 10.9% to 17.1%. The cost collapse is real across the mid-difficulty languages and nearly absent on the one language that carries the money. Buyers who read the $0.77 figure as "frontier-adjacent legacy modernization for pennies" will get a model that cannot finish the job.
Two caveats deserve their own paragraph, because both cut against the numbers. The first is that these are OpenAI's own evaluations of OpenAI's own models, with no independent run yet. The mitigation is that the company released the MultiLangBench setup and per-language results on GitHub as openai/MultiLangBench, which makes replication cheap for anyone who doubts the 5.4% baseline — and the baseline is the claim that does most of the rhetorical work here. The second is the saturation point: 97.9% on a self-designed test means the test has stopped discriminating between models, not that migration is finished.
The gap between a passing eval and a production migration is also wider than the numbers suggest. OpenAI says plainly that applying this to production software still requires expert review and testing against a system's business rules and dependencies, and the hidden tests only cover observable program output. A real COBOL estate includes job control, transaction monitors, indexed files, schedulers and batch dependencies that no migration eval touches. The program can be perfectly translated and the system still fail on Monday morning.
What the numbers do change is the economics of the rewrite business. Mainframe modernization has been sold for two decades as a labor problem — a shrinking pool of COBOL programmers, charged by the hour, against a backlog measured in billions of lines. If a model can get six of seven programs byte-exact at $2 per million input tokens before a human reviewer sees them, the billable premise weakens and the value shifts to verification: the firm that can prove a migration preserved behavior, under audit, is the one that gets paid.
What to watch: whether anyone independently reproduces the 5.4% GPT-5.6 Sol baseline on the released suite, and whether a bank or insurer puts a model-generated COBOL migration through an actual regulator's change-control process. That second event, not a benchmark run, is when this stops being a demo.
If a model can translate your COBOL byte-for-byte but not tell you which batch job depends on it, would you sign the migration — or hold the mainframe for another decade? Write to us via our contact page.
