Emergency
If storage is not responding and virtual machines are stalled, the person who answers is the one who works on it — including clusters we did not install.
The situations where calling beats waiting.
HEALTH_ERR that will not clearClusters often recover from HEALTH_WARN on their own. From HEALTH_ERR almost never: it means something is stopping I/O from proceeding, not just from being optimal. The longer it stays there, the less room is left to absorb the next event — a disk, a node.
peering or active+undersizedIn peering, those placement groups serve no I/O until the OSDs involved agree on which copy is authoritative — if it drags on for more than a few minutes, usually one OSD is answering intermittently. active+undersized means the data is reachable but with fewer copies than intended — a second failure at that point has no safety net left.
Flapping is almost never the disk’s fault. Most of the time it is the network: a mismatched MTU between nodes, a link dropping packets under load, or the cluster and public networks sharing the same cable. Every flap forces a rebalance, which in turn adds to the load that caused the flapping.
nearfull or fullAt full, Ceph blocks writes rather than risk filling an OSD completely, and the pool with it. It is not an alarm to put off: getting from nearfull to full can take just a few hours if a rebalance is already under way, and at that point even freeing space needs care not to make things worse.
A slow recovery is not an emergency on its own, but it becomes one if the cluster stays exposed in the meantime — fewer copies around, less margin for the next failure. The usual causes are a recovery bandwidth limit set too low, a saturated network, or disks that cannot keep up with ordinary writes on top of the recovery traffic.
Without monitor quorum the cluster cannot make decisions: the OSDs stay alive but nothing can tell them what to do, and I/O stops. The usual causes are two nodes lost out of three, clocks out of sync between the monitors, or a network partition isolating them in pairs.
Half a cluster on one version and half on another is tolerated by design — upgrades are meant to be gradual — but not indefinitely, and some features stay disabled until everything is aligned. What you should not do is roll back the components already upgraded: you go forward, in the right order.
The cluster keeps serving data without that node, but every extra minute is a minute without its copies. Before forcing a rebuild it is worth understanding why it is not coming back — a disk, the network, a boot stuck — because starting from the wrong point can cost more time than it saves.
Ceph absorbs failures that would stop any other storage system, but once it degrades the decisions have to be taken in the right order: forcing things in the wrong sequence can lose the very data you were trying to save. This is where experience counts for more than any manual.
There is only one first question: is the cluster still serving I/O or not. It changes everything about what can be done without making it worse.
Data comes back first, understanding why comes after. Every step is measurable and reversible: no commands that cannot be undone until the cluster is stable.
Once the cluster is stable we trace back what broke it — network, disks, CRUSH rules, a bad upgrade — and fix it, so it does not happen again in two months.
It is the most common case. You do not have to explain how it was built or why: that part is on us. We are often called in on clusters put together by someone who has since moved on.