13.1.2.5 CI Repeatable Tests

CI repeatable tests produce the same result every time they're run against the same code, a property that depends heavily on disposable, consistently configured environments and properly isolated test dependencies, since a test influenced by leftover or shared external state cannot reliably claim this property.

Why Repeatability Is Essential for CI to Be Trustworthy

A test suite that sometimes passes and sometimes fails against the exact same code, for reasons unrelated to the code itself, undermines confidence in CI as an actual verification gate.

docker compose -f docker-compose.test.yml run --rm app npm test

Running this exact command repeatedly, against the same code, should reliably produce the same result every time — any variability here points to an underlying repeatability problem worth investigating.

Common Causes of Non-Repeatable Test Behavior

Tests depending on shared external state, relying on specific timing assumptions, or depending on test execution order are common sources of non-repeatable behavior.

test('creates a user', async () => {
  const user = await db.users.create({ email: 'test@example.com' });
  expect(user.id).toBe(1);
});

A test asserting a specific, hardcoded ID like this assumes a particular starting database state — if run against a database with pre-existing data, this specific assertion could fail unpredictably depending on what data happens to already be present.

Using Fresh, Isolated Dependencies for Each Test Run

Providing a freshly created database (and other dependencies) for each test run, rather than reusing a persistent, shared one, removes a major source of this kind of unpredictability.

docker compose -f docker-compose.test.yml up -d test-db
docker compose -f docker-compose.test.yml run --rm app npm test
docker compose -f docker-compose.test.yml down -v

Diagnosing an Intermittently Failing Test

Running a suspected flaky test repeatedly, in isolation, against a freshly provisioned environment each time, helps confirm whether the issue is genuinely about repeatability or stems from something else entirely.

for i in {1..10}; do docker compose -f docker-compose.test.yml run --rm app npm test -- --testPathPattern=flaky.test.js; done

Why CI Repeatable Tests Matter

Test repeatability is foundational to CI actually serving its intended purpose as a reliable verification gate, and disposable, properly isolated environments are one of the most effective tools for achieving and maintaining this important property.