Running what we build
How a fourteen-person studio covers on-call for the systems it hosts.
We host most of the systems we build, so someone at Northfold is always on call. With fourteen people that means each engineer is on call about one week in eight.
Alerts go to the engineer on call first and to the system's named engineer second. Most alerts are handled in under fifteen minutes, and every alert outside working hours gets a short write-up the next morning.
The write-ups are the useful part. Each one ends with a change that makes the same alert less likely, and that change goes into the next release.