Ceph emergencies
The state names look alike and mean very different things. peering often clears itself, incomplete almost never: the first thing to establish is which of the two you have, because waiting is right in one case and expensive in the other.
Read them with ceph pg stat and ceph pg dump_stuck.
peering: they are agreeingThe OSDs involved are working out which copy is authoritative. For those seconds they serve no I/O, and that is normal: it happens every time an OSD restarts. It becomes a symptom if it lasts more than a few minutes, and then there is almost always an OSD answering intermittently rather than being clearly alive or clearly dead.
active+undersized: working, but exposedThe data is reachable and the VMs notice nothing, but there are fewer copies than intended. It is not an emergency the moment it happens: it becomes one because the next failure has no safety net. The cluster is already rebuilding on its own, if it has somewhere to put the copies.
degraded: copies missing, recovery runningSimilar to undersized but it says more: Ceph knows which copies are missing and is recreating them. The question to ask is whether the recovery is progressing or stalled: a degraded count that has not moved in hours is hiding another problem.
incomplete or down: not this oneCeph is saying it does not have enough information to serve that data: the copies it needs are on OSDs that are not there. These PGs do not clear by waiting, and they are the only ones where a wrong move genuinely loses data.
Three commands that tell you whether to wait or to act.
ceph pg dump_stuck lists only the stuck ones, with how long they have been stuck. A list that shortens by itself is a cluster doing its job; one sitting at the same number for half an hour is not.
ceph pg <id> query on a stuck PG names the OSDs it is waiting on. That is the question that matters: if those OSDs are powered off but recoverable, the answer is to bring them back, not to declare them lost.
In ceph -s, the recovering and misplaced numbers read twice a few minutes apart. If they are falling, the cluster is sorting itself out and the right thing to do is nothing.
Three of the four situations are resolved without any irreversible command.
It is nearly always the same problem as flapping: an OSD coming and going stops PGs from completing peering. Identify it from ceph pg <id> query, then find out why it answers intermittently, usually on the network. Taking it down for good (ceph osd down) is often better than letting it oscillate: the cluster stops waiting for it and rebuilds.
If recovery is progressing, the right move is to leave it alone and check there is room to rebuild into: a cluster near full has nowhere to put the missing copies and stays degraded indefinitely. If recovery is slow, raise the bandwidth limits with osd_max_backfills, one step at a time, watching that client I/O holds up.
The simplest answer and the most overlooked. An OSD that will not start because of a filesystem to check or a stopped service still holds the copies you need: recovering it closes the PGs and loses nothing. It is worth persisting with this before any other move.
incomplete PGs: this is a decision, not an executionThere are tools to unstick them, from ceph-objectstore-tool to export and reimport a PG from an OSD that will not start, through to declaring the missing copies lost. They work, and some cannot be undone. Which one to use depends on what is still on those disks, and that is established beforehand, not afterwards. If you have reached this point, write to us before running anything: the order matters more than the command.
Every restart puts more PGs back into peering and removes copies exactly when copies are short. If an OSD is answering badly it needs stopping, not restarting in circles.
osd lost because PGs will not moveIt tells Ceph to stop waiting for that OSD’s copies and accept what remains. It genuinely unsticks them, and if the OSD was recoverable the data only it held does not come back. It is the last move.
A pg_num that is too low is a real problem, but raising it now triggers data movement on a cluster already struggling to hold the copies it has. Do it afterwards, on a clean cluster.
A few minutes is normal, especially after an OSD or a node restarts. Beyond ten minutes on the same PG it is no longer waiting: something is stopping the OSDs from agreeing, and in most cases it is an OSD answering intermittently or a network dropping packets between two nodes.
You can, and it is often the right call: the cluster is rebuilding on its own. What to verify is that the recovery really is progressing and that there is room to rebuild into. A cluster that stays undersized for days is not waiting: it is stuck, and nobody notices until the next part fails.
Not necessarily, and that is exactly why it is worth stopping before acting. incomplete says Ceph cannot find the copies it expects, not that those copies no longer exist: very often they are on an OSD that will not start but can still be read. The moves that unstick things quickly are also the ones that make the loss permanent.
That is the usual case. You do not need to explain how it was put together or why: that is what we are there to work out.
Of all these symptoms, this is the one where haste costs most: the commands that unstick PGs quickly are the same ones that turn an avoidable loss into a permanent one. If it is not clear what is still on the missing OSDs, write to us before running anything.
Related symptoms: flapping OSDs,HEALTH_ERRAll the emergency symptoms
