Server health
Server health is the workflow for the thing nobody checks until it breaks. The decisions that stay with us are what to fix, and when. Checking every morning, and noticing when something is wrong, belong to the server itself and a watcher. This page shows how we used to check, how it runs now, and the parts it is built from.
How we built ours
The server reports on itself every morning, in writing. A check nobody has to remember is a check that happens.
How we used to do it
- We'd log in to the server when we thought of it.
- Check the disk, the memory and the load.
- Check that the web server, the database and each app were running.
- Check that last night's backup had run and been copied off the server.
- Most days everything was fine, so the checking drifted.
Quiet weeks between checks, and a problem found when something stopped.
What changed
We still decide what to fix and how. Remembering to look, and reading the numbers, stopped being ours.
How we do it now
- Every morning, after the backup, the server runs its own checks: disk, memory, load, the web server, the database and each app.
- It confirms the backup ran and was copied off the server.
- It writes the result to the board itself.
- The watcher reads the report and turns any failed check into an item in the morning briefing.
Nothing to read on a good day, and one line in the briefing on a bad one.
The parts
A check script on the server that runs after the nightly backup and writes its own row to the same board as everything else. The watcher in the morning run that reads it and raises anything that failed. And one design choice that makes it safe: the watchers can read and report but cannot touch the server, so any fix is proposed to us and made by us.
What it took us
The check script and its daily report came first, as an email. Writing the result to the board came with the September rebuild, and a second watcher, which grades each report and checks the backup against separate evidence, is partway through a trial of fourteen reports before its grading counts for anything.