Phase 10. Testing: the QA agent owns strategy and runs | Grigoriy Dobryakov

Grigoriy Dobryakov

Course · AI-Driven Development Lifecycle

Home ← Back to course
Phase 10ADLC course

Phase 10. Testing: the QA agent owns strategy and runs

Code has passed review and merged. Now comes verification that the product does what was promised: test strategy, cases, regression, a release-readiness verdict. The QA phase.

Testing is the phase where automation historically went furthest (automated tests have existed for decades), so it looks like there's nothing left to hand off. But running tests a human wrote is one thing; handing the agent the QA role itself — deciding what to test at all, deriving strategy from requirements, judging readiness — is another. This article is about that shift, and about the one weakness of the phase that automation doesn't cure, only stops hiding.

Ideation1 Discovery2 Marketing3 Requirements4 Architecture5 Formation6 Planning7 Implementation8 Code review9 Testing10 Release11 Operations12 Maintenance13 Governance14

The human role today

A QA engineer or SDET: builds a test strategy, writes cases and automated tests, runs regression, performs acceptance testing, delivers a verdict on release readiness. Classically this is a mix of manual and automated testing, test plans, bug reports, the role of quality gatekeeper before release.

Break it down. Generating test cases from requirements is mechanical work. Running them has long been machine work. Finding edge cases is combinatorics, where a machine is stronger. But one link is special: someone has to know what the correct answer should be. That's the oracle — and it's the weak point of testing in principle, not just for an agent.

What we hand to the agent

The QA agent holds the quality role. From prd/backlog with acceptance criteria, it derives a test strategy, generates cases from unit to e2e, runs them, finds defects and files them back to the board, and delivers a release-readiness verdict. It's not deciding "run the tests," it's deciding "is risk covered enough to ship."

The agent surpasses the human in volume and impartiality: it doesn't get tired generating edge cases, doesn't skip "boring" checks, and doesn't trim the test plan to fit a release deadline. Regression is free for it — it runs the full suite on every build, not "whatever we had time for."

Agent architecture

Phase state-machine

Inputs

prd/backlog with acceptance criteria (phase 4), architecture (phase 5), the built application after review (phase 9).

The agent holds the role

Tools: test-case generation; automated test runners; e2e and browser automation; fuzzing; coverage analysis; filing bugs to the tracker.

Artifact

a test strategy + a suite of automated tests + a coverage report + a release-readiness verdict.

Handoff: readiness verdict → Release (phase 11); filed defects → Planning and Implementation (phase 7 / phase 8). The release-readiness verdict as an artifact used to live as a human QA's judgment call. The agent makes it explicit and justified: not "seems ready," but "these risks are covered at this level, this is uncovered, here's my readiness assessment." The readiness judgment stops being opaque.

The release-readiness verdict as an artifact used to live as a human QA's judgment call. The agent makes it explicit and justified: not "seems ready," but "these risks are covered at this level, this is uncovered, here's my readiness assessment." The readiness judgment stops being opaque.

Where it breaks

The oracle problem. A test checks behavior against an expected outcome — but who said what the expected outcome should be? If both the requirements (phase 4) and the tests were written by an agent, a closed loop forms: the system checks itself against its own representation of "correct." Consistency isn't correctness. This isn't a weakness of any particular model — it's a structural hole shared with human QA; it just used to hide behind the tester's workload, and here it's exposed.

"Green tests ≠ a working product." The agent checks what it thought to check. Unknown-unknowns — risks no one thought to build into the strategy — are outside its field of view. And the "feel" of a product (is it comfortable to use, does it annoy) is poorly captured by a test case.

Accountability for a missed defect. A defect that reaches a user is a question of accountability. The agent widens coverage, but it doesn't become the one held responsible for a miss.

What stays human

An eyeballed acceptance of key user scenarios and — more importantly — supplying the oracle for critical spots: an explicit human "this is what's correct" where the cost of a mistake is high. A candidate for compression on generation and running; the durable remainder is the oracle for what's critical and accountability for a miss, and the latter traces back to the principal (phase 14).

human remainder ≈ 33%

Provocation / thesis

Testing is generating and checking hypotheses about system behavior against a spec — and the agent does all of it wider, faster, and more impartially than a human. The phase's weakness isn't in execution, it's the oracle: who defines the correct answer. And this isn't a new problem created by AI — it's testing's old hole, one that used to be masked by a human tester's limited attention and now comes into the open in an autonomous loop. What's non-transferable isn't testing, it's the oracle for what's critical — and that comes from a human at the points where being wrong isn't an option.

Running case

Vitrina in this phase

The QA agent builds an end-to-end scenario for Vitrina: a merchant assembles a storefront → a customer places an order → payment goes through → a notification arrives. It runs this on every build. And this is exactly where confirmation lands for what review (phase 9) sent back for rework: the QA agent runs the edge case of a repeated payment webhook — and verifies that after the idempotency fix, the order doesn't get duplicated. Regression coverage for the payment loop is now in the suite forever, free, on every build.

But note the boundary. The QA agent verifies that an order is "paid" according to how correct behavior is described in the requirements. If the very definition of "when an order counts as paid" had been wrong already in the PRD, the agent would have honestly tested the wrong spec and returned a green verdict. Here, the oracle for the payment loop is the one place where Vitrina's founder says once "this is correct" — and it's the single point in the phase that can't be handed to the agent. Artifact → Vitrina's test strategy + run, readiness verdict.

In practice

How it actually works — engineering breakdowns

Standalone howto from practice, showing this phase on real code and a working artifact.

Read next

Building AI-driven delivery in-house?

Designing the ADLC loop: where the agent holds the role and where the human principal remains — for your team and product.

Email me

The transition engine

Next Move Engine — the system that takes a team to an autonomous delivery loop.

Next Move Engine →