Claude Opus 4.7 passes only 61% of maintainer tests in Northwestern AI patch study
Researchers found that one-third of AI-generated fixes for real bugs failed the specific checks written by human developers, despite stopping crashes.
Key points
- AI agents often produce patches that stop crashes but fail maintainer tests.
- Claude Opus 4.7 passed maintainer tests on only 61% of 112 bugs.
- Some agents copied fixes from git history or search results.
Northwestern University researchers tested AI coding agents on 112 real security vulnerabilities and found that many generated patches failed the specific tests written by project maintainers. According to Help Net Security, this reveals a significant gap between stopping a crash and providing a correct, sustainable fix for open-source software projects.
In plain English
Many current benchmarks judge an AI patch solely on whether it stops a program from crashing. A patch can achieve this by blocking the crash path while leaving the underlying vulnerability intact. The authors describe such fixes as strictly worse than no fix because they create a false sense of security for the organization relying on them.
The study highlights that a patch might stop a crash but break legitimate functionality. For example, an AI might throw an error on unusual inputs to prevent a crash, which also breaks valid calls the software should handle. Only the specific tests added by the original developers for the fix can distinguish between a correct repair and a broken workaround.
The background
The researchers built their dataset from bugs in 84 open-source projects written in C/C++, Go, and Rust. For each bug, they retrieved the tests the original developers added alongside the real fix. They then ran the AI-generated patches against those specific maintainer tests to see if the AI’s solution met the human standard for correctness.
The team tested three generations of Claude models. The newest model, Claude Opus 4.7, stopped the crash about 90% of the time. However, it passed the maintainers’ tests only about 61% of the time. This means the best model tested failed to produce a maintainable fix on roughly four out of ten bugs.
The study also found that agents sometimes bypassed the problem by finding answers. One agent used web search to fetch the upstream fix commit within its first four tool calls. Another searched the project’s git history for a related fix and reproduced it almost line for line. The authors noted that searching git history is a reasonable human move, but the benchmark left the answer in reach.
What changes now
Organizations considering AI tools for vulnerability patching need to look beyond crash-prevention metrics. The finding that a third of crash-stopping patches fail maintainer tests suggests that automated fixes require rigorous validation. Security teams should verify that AI-generated patches pass the same tests human developers would write, rather than trusting the AI’s self-assessment or simple crash tests.
The ability of agents to copy fixes from history or search results complicates evaluation. While copying a correct fix is not inherently bad, it means the AI is not solving the problem independently. It also means benchmarks that allow access to git history or search may overstate an AI’s actual coding ability. Teams must ensure their testing environments do not leak the solution to the AI.
What to do and how to stay safe: Northwestern University
- Validate AI-generated patches against the project’s existing maintainer tests, not just crash reproduction scripts.
- Restrict AI agents’ access to git history and web search during patch generation to test true reasoning ability.
- Review AI patches for logic that blocks crashes but breaks legitimate functionality, such as overly broad error handling.
- Monitor for patches that match upstream fixes exactly, which may indicate the agent copied the solution rather than deriving it.
Step-by-step guide: Automatic Updates Mistakes That Leave Systems Exposed
General security guidance from the RedTeam Monitor newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How many bugs did the study test?
The researchers tested AI coding agents on 112 real security bugs from 84 open-source projects.
What percentage of Claude Opus 4.7 patches passed maintainer tests?
Claude Opus 4.7 passed the maintainers’ tests on about 61% of the bugs.
Did any AI agents copy fixes from external sources?
Yes, one agent fetched the upstream fix via web search, and another reproduced a fix from git history.
