Ceph emergencies
HEALTH_ERR is not a fault: it is the label Ceph puts over a dozen different faults. The first step is not to act but to read which of the dozen it is, because the right answer for one is the wrong answer for another.
The distinction is not about severity but about what is happening to the data.
HEALTH_WARN: the cluster is workingSomething is not optimal but I/O is flowing, and the cluster often recovers on its own: a recovery under way, an OSD that has just rejoined, space being rebalanced. Worth watching, not necessarily worth touching.
HEALTH_ERR: something is stopping itIt does not mean the cluster is slower: it means a condition is preventing it from proceeding, and that condition will not clear itself. From here a cluster almost never comes back without someone intervening.
The longer it stays in HEALTH_ERR, the less margin is left: the next event, a disk or a node, finds fewer copies and less room to absorb it. A cluster sitting in this state is not stable, it is waiting for the next failure.
The summary status tells you nothing useful. The command that matters is ceph health detail, and it is read from the top.
Not every line carries the same weight: the first [ERR] is the condition holding the cluster in this state, and the ones below it are often consequences. Fixing a consequence without the cause leaves everything as it was.
ceph health detail lists [ERR] and [WRN] together. Warnings underneath an error often clear themselves once the error is resolved: chasing them first is the most common way to lose time.
ceph -s tells you whether operations are blocked and for how long. A HEALTH_ERR with I/O flowing and one with I/O stopped are two different urgencies, even though the word is the same.
The same HEALTH_ERR, different causes, opposite answers.
OSD_FULL: one or more OSDs fullCeph has blocked writes rather than fill an OSD completely. You do not free space by deleting at random: raise the threshold temporarily by a hair to get I/O moving, free real space, then put the threshold back. Whoever raises it and forgets ends up in the same place with less margin.
PG_AVAILABILITY: placement groups not being servedThere is data no OSD is serving, usually because more OSDs are down than the pool tolerates. What matters here is working out whether those OSDs are recoverable before declaring them lost: an OSD that is merely powered off is one thing, an OSD marked lost is irreversible.
MON_DOWN or lost quorumWithout monitor quorum nobody decides anything, and I/O stops even with every OSD alive. The usual causes are two nodes lost out of three, clocks out of sync, or a network partition. Start from the monitors, not the OSDs: until there is quorum, acting on OSDs achieves nothing.
PG_DAMAGED: inconsistencies found by scrubThe scrub found copies that do not match. ceph pg repair often works, but first work out why they diverged: if the cause is a disk corrupting silently, the repair is temporary and the problem returns.
It is the first instinct and the most expensive. Every restart removes further copies exactly when copies are short, and a cluster that was degraded but stable can stop serving data altogether.
It buys breathing room to get I/O moving and free real space. Left raised, the threshold only moves the point at which the cluster will stop again, with less margin than before.
lost to get PGs movingIt works, and there is no going back: Ceph stops waiting for those copies and accepts what remains. If the OSD was recoverable, the data only it held is gone. It is the last move, not the first.
It is urgent rather than an emergency, and the difference is margin. The data is being served right now, but the condition that put the cluster there will not clear itself: the next failure will find fewer copies than it needs. Deal with it in the hours that follow, not when I/O stops as well.
It depends what caused it. An OSD_FULL with a rebalance under way can get worse within hours. A PG_DAMAGED can sit stable for days. The question to answer is not how much time is left but how many copies are: with one good copy, time stops mattering.
You can, and you should not. Some checks can be silenced with ceph health mute, which is useful to keep the others visible while working on one. Using it to turn the dashboard green means removing the only thing telling you what is going on.
That is the usual case. You do not need to explain how it was put together or why: that is what we are there to work out.
If you have read ceph health detail and the cause is not clear, or if I/O has already stopped, write to us before you act. In this state the order things are done in matters more than which things.
A related symptom: flapping OSDsAll the emergency symptoms
