How much test coverage do AI coding agents add to PRs?

How much test coverage do AI coding agents actually add to pull requests

AI coding agents add some test coverage, but not enough to assume the diff is protected. In recent studies, agent-written tests improved coverage in only a minority of PRs, and a large share of changed code still had no test execution at all. Existing tests also leave holes, so the right question is not whether an agent wrote tests, but whether those tests actually hit the changed lines.

The strongest recent evidence comes from an empirical study of 4,882 agent-generated pull requests in the AIDev dataset. The authors found that agents included test changes in only 49.6% of PRs that changed code under test files. That means the common case is not “the agent added tests,” it is “the agent shipped code, sometimes with tests, sometimes without them.”

Coverage from existing tests is uneven across languages. In the same study, existing tests covered 61.5% of changed executable lines in Java, but only 27.0% in Python. The Python result matters because it shows the agent is often working inside a weak safety net, not extending a strong one. A PR can look complete and still leave most changed lines untouched by any test.

Agent-written tests do help when they are present, but the gain is not universal. The study reports that coverage improved in 35.9% of Java code-plus-tests PRs and 22.5% of Python code-plus-tests PRs. The inconvenient part is that a test file in the PR does not mean the important lines are exercised. A test can raise the appearance of rigor while still missing the behavior that regressed.

Another recent study of 2,232 test-related commits from the AIDev dataset reached a similar conclusion from a different angle. AI-authored tests can produce coverage gains, and they sometimes look comparable to human-written tests in coverage impact. But that result is limited to commits where test addition was already happening, which is a narrower and more favorable slice than “all agent PRs.”

People often get this wrong by treating “the agent wrote tests” as the metric. The useful metric is whether the tests execute the lines that the PR changed, and whether they fail when the new behavior breaks. A green CI run can still miss the branch, exception path, or data edge case that the agent touched. Coverage is a map of attention, not proof of correctness.

The missed areas are not random. The agentic PR study found that error-handling constructs, such as try and catch blocks, were the most consistently under-tested, with miss rates reaching 86.0% in Java and 81.0% in Python. That is the inconvenient part for production work, because failures usually live in the exception paths, not in the happy path that the agent naturally prefers to exercise.

This is why coverage should be checked at the diff level, not only at the repository level. GitHub’s own agentic coverage workflow documentation describes an AI-powered agent that can open a pull request with a coverage workflow ready for review, and GitHub’s agentic workflow docs show automated PR review that checks whether changes are adequately tested. The workflow exists because human reviewers need a separate gate, not because the agent’s first pass is enough.

For a team using AI coding agents, the practical baseline is simple: require the PR to show which changed lines are executed by tests, not just that tests were added. If the agent changed parsing, validation, or error handling, inspect those paths explicitly. If the PR only adds a few assertions around the obvious success case, treat the coverage as partial, because that is usually what it is.

A concrete review pattern works better than a vague confidence check. Run the test suite with coverage turned on, compare the changed lines against the executed lines, and look for the gaps in branches and exception paths. If the agent’s patch is large, split it and ask for targeted tests per behavior. If the agent cannot produce those tests, the PR should not be judged as well covered just because it is green.

The summary is direct: AI coding agents usually add some test coverage, but the gain is inconsistent, often partial, and weakest where bugs are most expensive. They help most when a human reviewer forces the test to prove the diff, not when the agent is allowed to declare the diff covered on its own.

For DevConnect readers, that is exactly why test exchange beats blind trust. A teammate testing your PR on their own project catches the parts your agent skipped, and your review of their PR does the same in return. You can use DevConnect to coordinate that exchange at https://devconnectplatform.com, then still verify the changed lines yourself.

FAQ

Do AI coding agents usually add tests to every pull request

No. In the agentic PR study, test changes appeared in only 49.6% of PRs that changed code under test files. A PR can be perfectly valid and still contain no new tests, so review the actual diff instead of assuming test work happened.

Are AI-written tests as good as human-written tests

Not by default. One study found that AI-authored tests can contribute to coverage and may look comparable in some coverage outcomes, but that result came from commits already focused on test changes. Quality still depends on whether the tests exercise the changed behavior and fail for the right reasons.

What should reviewers check first on an agent-written PR

Check the changed lines that were actually executed by tests. Then inspect error paths, boundary conditions, and any logic the agent may have covered only through happy-path assertions. A green run without diff coverage still leaves room for regressions.

Is repository-level coverage enough to judge an agent PR

No. Repository-level coverage can hide untouched new code, especially in branches and exception handling. Diff-level coverage is the useful check because it shows whether the new lines in the PR were executed by tests.