The same OSD goes down and comes back up
In ceph health detail you see messages like osd.4 marked down after no beacon repeating over time, and the OSDs involved are always the same ones, or always on the same node.
Ceph emergencies
Flapping OSDs: it is almost always the network
An OSD that keeps leaving and rejoining the cluster rarely has a failing disk. On the clusters we are called to, the cause is almost always the network, and replacing the disk first wastes time while the cluster stays exposed.
How to recognise it
The cluster reports OSDs changing state on their own, with nobody touching them.
The same OSD goes
Recovery restarts every time
Each flap sets the cluster moving data again. In ceph -s the recovery never finishes: it starts over before it completes.
The disk reports no errors
SMART is clean, dmesg shows no I/O errors, and the disk answers normally when queried. That is the sign the problem is somewhere else.
The three causes we find most often
In order of frequency, on production clusters we did not install ourselves.
Mismatched MTU between nodes
One node set to 9000 and the others to 1500, or a switch that does not pass jumbo frames. Small packets get through and large ones do not: the cluster looks fine until traffic crosses the threshold, then heartbeats are lost and the OSD is declared dead.
A link dropping packets under load
A cable, a transceiver or a port that only degrades when traffic rises. At rest the link is perfect, and the very rebalance the flapping triggers pushes the load to the point where the link gives way: the problem feeds itself.
Cluster and public networks on the same cable
When replication traffic and client traffic share the link, one backup or one busy VM is enough to delay heartbeats past the threshold. Ceph reads the delay as a dead OSD.
What to check, in this order
Three checks that take minutes and that rule the network in or out before touching any disk.
The MTU, node to node
A ping with a large packet and fragmentation refused: ping -M do -s 8972 <other-node>. If 1472 gets through and 8972 does not, the MTU is inconsistent along the path. Test it between every pair of nodes, not just from one.
Interface errors
ip -s link on each node: dropped or errors counters that keep climbing point at the link, not the disk. Read them twice a few minutes apart, because what matters is whether they rise.
Which OSDs, on which nodes
ceph osd tree tells you whether the flapping OSDs all sit on one node or behind one switch. If they do, that network path is almost certainly the cause.
How it is fixed
Two of the three causes are sorted in minutes, with the cluster running. The third calls for decisions, not just commands.
MTU: align it across every node
One node at a time, with nothing stopped: ip link set dev <interface> mtu 9000 to test, then the same line in the network configuration so it survives a reboot. The switch has to pass jumbo frames on every port involved, otherwise the problem moves rather than goes away. After each node, repeat the large-packet ping: if it gets through, that path is clear.
A link dropping packets: replace the part
Cable, transceiver, then switch port, in that order of likelihood. Set the node to noout, replace, clear noout, and watch the ip -s link counters for a few hours under load. If they start climbing again, it was a different part: carry on to the next one rather than undoing the first.
Separating cluster and public networks: this one is a decision
The fix is well known (a dedicated network for replication, declared in cluster_network) but doing it on a cluster that is serving data is not a command, it is a sequence. Which node to touch first, whether to stop the rebalance or let it run, what to do if an OSD does not come back halfway through. Getting the order wrong on an already exposed cluster costs more than the problem being fixed. If you are at this point and the cluster is already short of copies, write to us before you start: half an hour with someone who has done it costs less than a recovery gone wrong.
What not to do
Replace the disk first
If the network is the cause, the new disk will flap like the old one. Meanwhile the cluster has lost another copy, and the rebalance adds load to the very link that is giving way.
Raise the thresholds to silence the alarms
Stretching the heartbeat timers makes the symptom disappear and leaves the problem. A cluster that will not declare a genuinely unreachable OSD dead stops protecting the data, which is the only thing it exists to do.
Set noout and forget it
Setting noout is the right move to stop the rebalance while you investigate, and it is the first command we run too. It becomes a problem if it stays set: the cluster stops reacting to real failures, and the next one is not absorbed.
Common questions
Can flapping really be the disk?
It can, and it is the least frequent case. It happens with disks that answer at very high latency without reporting errors: the OSD does not die but fails to answer heartbeats in time. You can tell because the OSDs involved are spread across different nodes rather than concentrated on one, and because the latencies are high even on an idle cluster.
How urgent is it?
It depends how many copies are left. With three replicas and one flapping OSD the data is still protected, but every flap burns bandwidth and lengthens the recovery. With two copies, or with a second OSD already out, it is an emergency: there is no margin left for the next failure.
Can it be fixed without stopping the cluster?
Almost always. Correcting an MTU or moving the cluster network onto a dedicated link is done with the cluster running, one node at a time. Stopping everything is rarely needed, and when it is, it is because the flapping has already taken placement groups out of service.
Do you work on clusters you did not install?
That is the usual case. You do not need to explain how it was put together or why: that is what we are there to work out.
If it is happening right now
Get in touchA flapping OSD does not stop the cluster, but it leaves it exposed for as long as it lasts. If you cannot isolate the cause, write to us: we have seen these before, including on clusters we did not install.
