One failing job in our nightly matrix cancelled the eleven that would have told us more

Our database client library supports four Postgres versions and three Python versions, and every night a GitHub Actions workflow runs the integration suite across all twelve combinations. In the week before the 3.0 release it failed five nights in a row, and every morning it reported exactly one broken job. On Monday it was Python 3.10 against Postgres 13, a test matching the exact text of an error message that 3.10 had reworded. We fixed it. On Tuesday the same workflow failed on Python 3.12 against Postgres 16, a test helper still importing distutils, which 3.12 removed. Wednesday was a third failure, Thursday a fourth. None of those failures was new. All of them had been there on Monday. A matrix in GitHub Actions has fail-fast set to true unless you say otherwise, which means that when any one job fails, every job in the matrix that is still queued or running is cancelled. Our first failure arrived about four minutes in, while most of the other eleven were still installing dependencies. They never reached their tests. Their conclusion was cancelled, which the summary page draws in grey, and our Slack notifier only posted jobs whose conclusion was failure. So each morning we saw one red line, fixed it, and released the next failure from behind it. For a pull request that default is reasonable. Somebody is waiting, one failure is enough to send the change back, and stopping early saves runner minutes. A nightly run exists for the opposite reason. Nobody is waiting, and its whole value is the complete list of what is broken. The nightly workflow now sets fail-fast to false, and the pull request workflow keeps the default on purpose. The notifier reports every conclusion, not only failures, and a night with any cancelled job is posted as incomplete rather than as one failure. When we turned the nightly run back on with the new settings, it reported the last two problems at once, and the release went out the following Tuesday. A cancelled job is not a passed job and not a failed one. It is a question nobody asked, and for a week we read the silence as an answer. – Sergey Shinder