Contents

Why do we run end-to-end tests against a real database?

Because a mock confirms the assumption you had while writing it, and a real database contradicts it. The failures that get expensive in distributed systems sit exactly where the mock knows nothing: in transaction boundaries, isolation levels, the ordering of migrations and the time zone behaviour of timestamps. A test against an in-memory substitute measures the substitute, not the system.

In EcoPulse the end-to-end tests of the REST services therefore run against a genuinely started API with a real database — the same PostgreSQL version as in production, the same migrations, the same constraints. That is not perfectionism; it is the consequence of findings that would not have surfaced otherwise.

What exactly does Testcontainers start — and what does it not?

Testcontainers brings up real containers before the test run — PostgreSQL with TimescaleDB, Redis, a Kafka broker — and hands the test the actual connection details. Afterwards they disappear again. What it cannot do: devices. An inverter, a wallbox or a PLC exists in no CI. That limit shapes the entire test architecture.

From it follows a split that has proven itself for us:

  • Infrastructure is genuinely started. Database, cache, broker — anything that can be a container is a container.
  • Third-party devices are replaced by recorded telemetry, not by hand-written mocks. The difference is substantial: a recording brings the outliers, gaps and duplicate messages that actually occur in the field. A hand-written mock brings whatever its author thought of.

What does it cost in pipeline runtime?

Honestly: time. A container takes seconds to start, and doing that per test class quickly produces a pipeline nobody waits for. A pipeline nobody waits for is worthless as a quality gate, because it gets bypassed.

The cost is real but controllable. The mistake is not starting real dependencies. The mistake is starting them too often.

How do you keep the runtime manageable?

Through four measures that together make the difference between ten minutes and an hour:

Reuse containers per test suite, not per test method. Between tests the state is reset, not the container restarted. That is the single biggest lever.

Separate test levels. Unit tests run on every commit in seconds; the container-backed integration and E2E tests run in the merge gate. Putting both in the same run penalises every commit with the runtime of the most expensive test.

Parallelise honestly. Tests run in parallel when they are genuinely independent — and that has to be verified, not assumed. Two suites sharing the same database are not independent, even if they look like it on the first run.

Serialise heavy jobs. In one of our projects four parallel feature pipelines stretched a 25-minute run to over an hour; two jobs died in the runner’s time limit. Four serial runs that complete are cheaper than four parallel ones that all fail. In GitLab CI a resource_group does this.

How do you test authorisation paths?

By treating them as test cases of their own, not as a side condition. The usual mistake is to run every test as an authorised user and assume the permission check will hold. It holds exactly until someone adds a route and forgets the check — and then no test suite reports anything.

We therefore check at least three cases per endpoint: authorised, unauthorised, unauthenticated. In multi-tenant systems a fourth is added, the most important one: authorised, but for a different tenant. That case separates working tenant isolation from isolation that merely looks like it — and it can only be checked against a real database, because it depends on data that lives there.

How do you test error paths?

By actually producing the failure instead of simulating it. With real containers that works: the database is stopped mid-run, the broker loses its connection, a migration runs against an unexpected data set. Those are the states in which it is decided whether a system fails cleanly or loses data.

Two classes are especially worthwhile:

  • Recovery. What happens when a service restarts after an outage and the event stream has moved on? For an event-driven platform that is not an edge case but normal operation.
  • Partial outage. The cache is gone, the database is up. Does the service then answer more slowly — or wrongly?

A status code alone is not a statement here. In another project four backend routes silently returned HTTP 200 with the content of the single-page app instead of the API response; the client checked IsSuccessStatusCode, which is true at 200, and happily reported success. Since then our error-path tests check status code and content type.

What about streaming and time series data?

That is the part standard test tooling does not cover. A time series analysis is only tested once it copes with irregular intervals, gaps, duplicate timestamps and measurements arriving retrospectively — and no mock delivers any of that, because nobody thinks of it.

We bring up TimescaleDB as a container and replay recorded telemetry into it. What is checked is not only the result but its stability: the same analysis over the same period must return the same value, regardless of the order in which the measurements arrived.

Where does the test data come from?

From synthetic data sets derived from real recordings. In public administration projects that is mandatory — real data is ruled out. In industrial projects it is the better choice anyway, because a synthetic set is reproducible and a production dump is not.

What matters is carrying the quirks over rather than smoothing them away: a test data set without outliers checks a system that does not exist.

When is Testcontainers the wrong tool?

In four cases we reach for something else:

  • When the dependency cannot be a container. A PLC, an inverter, a telephone endpoint. For Phonalisa we instead operate dedicated test environments for signalling and media path — real hardware, because voice quality cannot be containerised.
  • When the business logic is only reachable through the interface. Then the route is GUI automation: Squish for Qt applications, Selenium for web interfaces. Both are more expensive to maintain and therefore come last.
  • When a unit test is enough. Checking a pure calculation against a database lengthens the pipeline without adding insight.
  • When the environment does not permit containers. In some regulated environments, virtual machines via Proxmox or VMware are the prescribed route.

How do you start when none of this exists?

Not with the broadest level, but with the most expensive class of defect. In practice that almost always means: one endpoint that writes data, checked completely against real infrastructure — success case, unauthorised, unauthenticated, foreign tenant, and one error path. That is five tests, standing within a day and protecting something from the first run onwards.

After that the suite grows along change frequency, not along the folder structure. Wherever most commits landed in the last six months is where the risk sits — and that is where coverage pays off first. The opposite route, covering everything broadly first, produces a long runtime without a statement about the critical paths.

One point belongs settled from the start, because it can barely be retrofitted: the suite has to run in the merge gate, not beside it. A test suite that stops nobody gets switched off at the first red light rather than repaired.

How do you know a test really protects something?

By the fact that it fails the moment you take the safeguard back out. “Test added” says nothing about quality. The proof reads: mutation introduced, test red, mutation reverted, test green. Only then is it established that the test checks what its name claims.

That check is uncomfortable and finds a surprising amount — including in tests that had been green for months. Described in detail in the field report How we develop with AI agents, where the same proof is mandatory for agent-generated regression tests.

The rule we turned this into

Tests against real infrastructure are mandatory in the merge gate. What runs against mocks is a unit test and may not call itself E2E. The distinction sounds pedantic, but it decides how much trust a green pipeline deserves — and trust in the pipeline is the precondition for it being accepted as a gate at all.

How we introduce this into existing test estates is described under Quality assurance & test automation. Whether the build belongs in house or outside is covered in Build test automation in house or contract it out?