Failover
When a node dies, Clouduler moves its containers to other nodes. It also makes sure the dead (or cut-off) node stops its own containers first, so one volume never has two writers. This page explains how the cluster decides a node is gone, what happens step by step, and the timers you can tune.
Raft and quorum in plain words
The cluster's database (stacks, assignments, volumes, config) is kept identical on every node by an algorithm called raft (glossary). A change is accepted only when a majority of the voting nodes has it. Voters are the managers. Workers receive every change and run containers but never vote. Having a majority is called having quorum. Without quorum the cluster pauses changes rather than risk two nodes disagreeing.
Quorum is managers / 2 + 1, rounded down:
| Managers | Needed for quorum | Managers that can fail |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |
| 7 | 4 | 3 |
Workers do not change these numbers. Use three managers as the minimum for tolerating a failure. Two managers are worse than one: if either dies, the survivor loses quorum and, after the fence timeout, stops all its containers. Nothing warns you about an even number of managers.
How it works: two halves
- Fencing (every node, about itself). A node that has had no quorum for
fence_timeoutstops the containers it runs. - Rescheduling (the raft leader, about other nodes). The leader pings every node each
probe_interval. A node silent for longer thannode_timeoutis marked unreachable, and the scheduler assigns its instances to other eligible nodes.
The timers are set so the fence fires first: the cut-off node gives up its containers before the rest of the cluster starts new copies.
What happens when a node dies
Assume worker cirrus loses power or network. Times use the defaults and are estimates, not measurements.
- Quorum lost on cirrus. Within about 1.5 seconds it notices it cannot hear the leader.
- Fence clock starts. Nothing is stopped yet. A brief blip that recovers resets the clock.
- Leader probes fail. Once no ping has succeeded for
node_timeout(60s), the leader marks cirrus unreachable (log:unreachable, rescheduling its containers). - The scheduler moves the work. Each instance goes to the least-loaded node that passes every check: schedulable, reachable, holds a usable copy of each replicated volume, has the networks and ports needed. See Placement.
- Volumes follow. The new node becomes the volume's primary and its generation increases.
- The new node starts the containers, pulling the image first if needed.
- Cirrus's fence fires about 16 to 22 seconds after the cut and stops its containers. A container that stops on SIGTERM is down in a second or two; one that ignores it is killed at its grace period.
Instances stay where they land. When cirrus comes back, nothing moves back automatically. Use Rebalance if you want that.
- Web UI
- CLI / config
Open Nodes (/nodes). A dead node shows Unreachable in the State column, and a banner says "NAMES is unreachable. Its containers move elsewhere once the failover timeout passes." A node without quorum shows the No quorum banner: "This node can reach X of Y voters, so changes are paused. Bring a manager back online."
The Overview flags the unreachable node and any stack that could not be placed (here machine-learning: "no eligible node: boreas is unreachable and cirrus has too little free memory"), and shows 5 of 9 containers running.
clouduler status # leader, quorum, each node "UNREACHABLE" or "up", problems
clouduler stack unhealthy --all # what is not running as asked
clouduler stack unhealthy --placement # why instances cannot be placed
Placement reasons you may see: NODE is unreachable, NODE is Drain, NODE is not the pinned node PIN, NODE holds no current copy of volume VOLUME, NODE does not implement drive DRIVE.
Data you can lose
Replicated volumes reach other nodes only on their sync interval (default 15 minutes). A primary that dies takes its unsynced writes with it, and failover starts the service from the freshest copy that exists. See Drives and volumes. Pinned services (x-clouduler.node) are not moved and wait for their node. An instance with nowhere to go stays assigned to the dead node and a cannot place instance warning is logged.
Restarts and returning nodes
- A daemon that restarts within
node_timeoutof its last quorum keeps its containers running while it re-establishes quorum (state Hold). After a longer gap it starts Closed: it stops containers until it sees quorum. - When a node returns, it removes any container no longer assigned to it (the ones that moved away). Their volume data stays on disk.
- A new leader waits a full
node_timeoutbefore declaring any node dead. - A node that was removed while down wipes its keys and exits on its next start:
removed from the cluster while down; local secrets wiped, re-provision to rejoin.
Tune the timers
- Web UI
- CLI / config
Open Settings, General (/settings/general), section Failover. Fields: Declare a node lost after, Stop containers without quorum after, Check nodes every, Container stop grace period (hint "Applies live."). An alert appears when the margin is too small.
failover:
node_timeout: 60s # leader gives a silent node's work to others after this
fence_timeout: 15s # a node without quorum stops its own containers after this
probe_interval: 5s # leader probes every node this often
stop_grace: 30s # default time a container has to stop before SIGKILL
clouduler config reload
sudo systemctl restart clouduler.service # needed for the three timers
A service can set its own stop_grace_period: 2m in the stack file.
Rules: probe_interval < fence_timeout < node_timeout, all enforced as errors.
The fence margin warning
The defaults (fence_timeout 15s, stop_grace 30s, node_timeout 60s) leave a margin, so a stock config logs nothing. The daemon (and clouduler check, and the UI alert) warns when fence_timeout + stop_grace >= node_timeout:
failover.fence_timeout (15s) plus the default stop grace (30s) reaches node_timeout (45s): a fence may still be stopping containers when the leader reschedules them, giving one volume two writers. Raise node_timeout or lower fence_timeout or the stop grace
It is a worst-case warning, not a sign that something is broken: it assumes a container uses its full stop grace. It matters if a container ignores SIGTERM, because it can still be running (and writing) when the new copy starts. To silence it, raise node_timeout or lower fence_timeout or stop_grace. If a service has stop_grace_period: 2m, node_timeout needs to be at least about 2m15s for the daemon to stay quiet; the daemon checks the longest per-service value too (that per-service check needs the deployed stacks, so only the daemon log and UI report it, not clouduler check). The warning ignores a further 10 seconds of slack a fence stop may take. A longer node_timeout also makes real failover slower.
Options
durationdefault 60sdurationdefault 15sdurationdefault 5sdurationdefault 30sdurationdefault failover.stop_graceThe failover block rejects unknown keys, so a typo fails config loading with unknown field. Fixed internals you cannot change: quorum is re-checked every 500 ms and requires leader contact within 1.5 s; raft heartbeat 500 ms; scheduler sweep 10 s.
Quirks and gotchas
The failover timers node_timeout (60s), fence_timeout (15s) and probe_interval (5s by default) are read when the daemon starts; stop_grace (30s) is live. The leader's failure detector uses a 500 ms heartbeat and a 1.5 s quorum acknowledgement window, and the scheduler sweeps every 10 seconds.