Field Notes
Flaky CI: separate timing, shared state, and test design
Restarting the job is not a strategy. Sorting flake categories is how ownership becomes possible.
Flaky builds exhaust teams because every failure feels the same in Slack. They are not the same in the pipeline.
Three buckets that help
- Timing — races, sleep-based waits, eventual consistency assumptions
- Shared state — a staging database, a shared queue, a singleton test user
- Test design — assertions on wall-clock UI details or order-dependent suites
Make a triage board
In a CI Reliability Clinic we ask the room to place the last twenty failures into those buckets before debating tools. Ownership follows the bucket: application squads often own test design; platform often owns shared runners and caches; product owns whether a shared staging database is still acceptable.
Resist the new runner fantasy
Buying more capacity without naming flake categories usually buys quieter afternoons for a fortnight. The queue returns. The categories remain.