Contents
Why is the question framed wrongly?
Test automation is not a project with an end but an estate with a maintenance load. The question is therefore not “who builds it” but “who maintains it in three years”. Skip that and you buy a test suite that is red after two releases, then skipped, then switched off — and you pay for it twice.
How do the two routes compare?
| Criterion | In-house build | Contracted out |
|---|---|---|
| Ramp-up time | Long. Tool selection, learning curve and the first discarded attempts included | Short. Experience is bought in, the first tests run within days |
| Product domain knowledge | Present. The biggest advantage, and often underestimated | Has to be built up, costs the first few weeks |
| Tool and method knowledge | Has to be built up | Present |
| Short-term cost | Lower, but hidden: developer time missing elsewhere | Higher and visible on the invoice |
| Long-term cost | Lower, if the competence stays in house | Ongoing, as long as it is maintained externally |
| Day-to-day prioritisation | Regularly loses against the next customer ticket | Protected, because it is commissioned and scheduled |
| Test depth | Grows slowly, but fits the product precisely | Grows quickly, but can miss the product’s real risks |
| Maintainability after handover | High, because the authors sit in house | Depends on whether knowledge transfer was part of the assignment |
| Typical failure | “We’ll do it on the side” | Handover without knowledge transfer |
Where does the in-house build fail?
Almost always at the same point: test automation runs beside day-to-day work and loses every prioritisation against a customer ticket. After one quarter there are twenty tests, after two, five of them are red, and nobody has time to repair them.
The second classic is tool selection without experience. A framework that does not fit the application — GUI automation for a Qt interface with a tool that only knows the DOM — costs more later than the entire build. That decision is made at the start, when knowledge is at its lowest.
The third is missing anchoring. Tests that do not run as a merge gate are documentation, not a safeguard. As long as a red test stops nobody, it does not get fixed.
Where does contracting out fail?
At the handover. A suite created externally that nobody owns internally does not get adapted at the first major rebuild — it gets disabled. The fault rarely lies in the suite; it lies in knowledge transfer not being part of the assignment.
The second reason is missing product understanding. An external supplier tests what is specified. The interesting defects, however, sit where the specification is silent: in special cases, historical data and the one constellation only a single customer has. Your team knows those cases.
Which hybrid works?
Build externally, continue internally — under three conditions that are not negotiable:
Together from the start. At least one person from your team works along during the build phase, not only at handover. Knowledge transfer at the end of a project is an appointment, not a transfer.
Choose tools for your landscape, not for the supplier’s preference. If your team writes C#, a suite in another language is a handover brake — however elegant it may be.
CI anchoring is part of the delivery. Tests that do not run as a merge gate do not count as delivered. That belongs in the statement of work, not in the hope.
What does a test suite cost to maintain?
The number that appears in no quotation. In our experience the ongoing maintenance effort amounts to a noticeable share of the build effort per year — driven by changes to the interface, to APIs and to test data.
Two levers reduce it considerably:
- Test against stable interfaces rather than the interface layer wherever possible. An API test survives a redesign, a GUI test does not.
- Real dependencies instead of mocks at the integration level. A mock has to be updated with every change; a database started via Testcontainers updates itself. How we cut this is described in E2E tests against real databases with Testcontainers.
Which test levels pay off first?
Not the most visible ones. The order that has proven itself for us:
- Integration tests against real dependencies. They find the defects that actually occur — transaction boundaries, migration ordering, time zone behaviour — and are surprisingly stable, because they do not hang off the interface.
- Unit tests in the functional core. Fast, cheap, but only valuable where logic actually sits. Testing getters raises the coverage number and nothing else.
- E2E tests for the critical paths. Login, the ordering flow, the one process whose failure gets expensive. Deliberately few — they are the most expensive to maintain.
- GUI automation last, and only where the business logic is reachable exclusively through the interface. In administrative systems with form logic that is the norm; in an API-driven application it is the exception.
Starting the other way round — GUI first, the rest later — leaves you after six months with a slow, unstable suite and no statement about the core.
How do you recognise a usable supplier?
By three questions they ask unprompted:
- “What runs in your CI today?” Anyone who does not ask is planning a suite that stands beside the pipeline.
- “Which defects slipped into production most recently?” That is the only solid basis for prioritisation. Asking instead for the target coverage number optimises a metric.
- “Who on your side takes over the suite?” If that question does not come, knowledge transfer is not planned.
Conversely, a warning sign: a quotation that guarantees a coverage percentage without having seen the code. Coverage is a by-product, not a goal — 90 % across trivial paths is worth less than 40 % across the places where it hurts.
What about AI-generated tests?
They shift the calculation, not the question. An agent writes tests quickly and in volume — and that is exactly the risk: a large suite of tests all running against the same wrong assumption is more dangerous than none, because it simulates safety.
What helps is the safeguard behind it. Here, agent-generated tests pass the same gates as hand-written ones, and a regression test only counts as real once it demonstrably fails without the corresponding fix. Without that proof a green test is merely an assertion. How we secure this is described in The guardrails we set for agents.
For the original question that means: AI significantly lowers the cost of building, and only slightly the cost of maintaining. The question “who maintains it in three years” is unchanged.
What is the usual entry point?
An assessment of the existing test landscape: what is covered, what is red, what is flaky, where the runtime sits, what actually runs in CI. That is a few days of effort and usually answers the original question by itself.
In practice the result is rarely “start over”. More often it is: stabilise a handful of flaky tests, halve the runtime, hang the suite in the merge gate — and only then decide who expands it. Details under Quality assurance & test automation.