Skip to main content

Node lifecycle

A node joins the cluster, runs containers, may be drained for maintenance, and is eventually removed. This page covers each step, plus rebalance, automatic crash-loop rescheduling and clear-errors. For unplanned failures see Failover.

Add a node​

Open Nodes (/nodes) and click Add node. In the Add a node sheet enter Node name, choose Worker or Manager, optionally an Address, and click Create join command. Copy the command under Run this on the new node and run it on the new machine. The key in the command is shown once; if you do not use it, delete the key under Settings, Access.

/nodes
Nodes. Rebalance cluster and Add node are top right.

A manager becomes a voter only after the leader pings it, so a dead node is never promoted. Remember the quorum rules: Failover.

Drain a node and set availability​

A node is active (can receive containers) or drain (nothing new is placed on it, and its containers move off).

Open the node page (/nodes/boreas) and click Drain boreas. Confirm: "Its containers move to other nodes and nothing new is placed here." To bring it back, click Activate boreas ("New containers can be placed here again.").

/nodes/boreas
Node detail with the Drain button.

What happens: on the next scheduling pass (within 10 seconds) each instance on the drained node is moved to another eligible node. The old container is stopped and removed, the new one starts. Activating does not move anything back. An instance with no other eligible node stays where it is, still running.

Instances that mount volumes are moved with the same pipeline as Move (stop, final sync, verify, switch, start), so no writes since the last periodic sync are lost. One migration at a time runs per instance; if a migration fails the instance stays on the drained node until you retry or dismiss it on its page. Instances without volumes, and instances on a node that is already unreachable, are simply re-assigned.

Remove a node​

On the node page, panel Retire this node, click Remove boreas. The button is disabled while the node still runs containers ("boreas still runs N containers. Drain it first."). Confirm "Remove boreas?" ("This takes the node out of the cluster for good. To use the machine again, run init on it.") and type the node name.

If the node is unreachable, failover reschedules its instances and removal then works. Stopped stacks still count as assignments. Removing a node also hands each volume it was primary for to the most up-to-date remaining replica, which may be behind: data written only on the removed node is gone, and removal does not check.

Rebalance​

rebalance moves already-placed instances to even out load. The normal scheduler never moves a healthy instance, and a returning node gets nothing back on its own, so this is how you spread work out again. It runs only on request.

On Nodes click Rebalance cluster and confirm ("Instances may move between nodes to even out load. Each move restarts that instance."). The last run appears under History.

It skips instances pinned to a node, global services, instances that share a proxy hostname or colocate group, ones already mid-move, and ones with only one eligible node. Load is memory pressure first (share of node memory taken by instances with a memory limit), then instance count. A move happens only if the gap is at least one percentage point (or counts differ by 2) and the target would not end up busier than the source was. Each move goes through the full move pipeline: stop, final volume sync, verify, switch, start. Moves use the cluster failover.stop_grace, not a service's own stop_grace_period, so a database with a long grace period can be killed mid-write; raise failover.stop_grace to cover your slowest service.

Crash-loop rescheduling​

An instance restarted 3 times within 120 seconds on one node is a crash loop. Instead of restarting forever, Clouduler moves it to the least-loaded eligible node it has not been moved away from in the last 30 minutes, through the same move pipeline. Log: instance keeps restarting here, moving it to another node.

If it cannot be moved (pinned, global, anchored by a proxy hostname or colocate group, or every node already tried), it restarts in place with a delay that starts at 30 seconds, doubles, and caps at 10 minutes (keeps restarting here but cannot be safely moved (pinned, global, or anchored), restarting in place). Redeploying with a changed spec gives every node another try.

A crash loop caused by a too-small memory limit does not get fixed by moving; the same limit applies everywhere. Raise the limit.

A failed move shows on the instance page with buttons Try moving to NODE again, Dismiss, Move to another node and Restart on NODE.

Clear recorded errors​

clear-errors forgets recorded failures so they stop being reported as problems: failed backup runs and restores, failed image updates, failed migrations, certificate and DNS errors, and alert destinations whose last delivery failed. Running, pending and succeeded records stay. A failure that is still happening is reported again by its next attempt. CLI only.

clouduler clear-errors      # cleared 3 failed backup runs, 1 failed migrations
                            # or: no recorded errors to clear

Clearing a failed move does not move the instance; it stays on whichever node it is on.

Who can reach whom​

The Nodes page has a "Who can reach whom" matrix. Every node probes its peers itself and reports what it sees, so a one-way break shows up as two different rows. Rows report, columns are peers.

  • The raft side is an authenticated raft ping on a fresh connection (the cluster-key handshake plus one request and reply), not a bare TCP connect. The figure is its round trip in milliseconds, handshake included. Probes time out after 2 seconds.
  • The API side is a TCP connect to the peer's web port. If raft works and this does not, the cell reads "API down".
  • Each node also times a 4 KiB write plus fsync in its data dir (disk_ms). A raft follower must fsync every append before it can answer the leader, so a node whose fsync takes a second or more answers pings yet cannot keep up. Cells pointing at it read "Slow disk (x.x s)", and it tends to flap in and out of quorum. The probe gives up after 8 seconds.
  • A cell reads "Marked down" when the link works from the reporting node but the leader has the peer on its unreachable list. "No report" means the row's node could not be asked, or did not list that peer.

To stop a node from running the disk probe (an SD card you want to spare, for instance), set nodes.<name>.disk_probe: false. That node then reports no disk figure and is never shown as "Slow disk". See cluster.yaml.

Options​

Option
Type / default
Reload
Description
node add NAME --role {manager|worker}required
commandno default
n/a
Writes a roster entry only.
Watch out: Also needs --rpc-addr HOST:PORT. Optional: --drive NAME=PATH (repeatable), --label KEY=VALUE (repeatable), --via NODE.
node remove NAME
commandno default
n/a
Removes a node that runs no instances.
Watch out: --via NODE optional.
node set-availability NAME {active|drain}
commandno default
n/a
Set whether the node can take work.
Watch out: --via NODE optional.
rebalance [--yes] [--via NODE]
commandno default
n/a
Plan and submit load-evening moves.
clear-errors [--via NODE]
commandno default
n/a
Forget recorded failures.

Changing a node's address​

You can change the internal subnet and every node's internal IP, managers included, by editing cluster.yaml once. Nothing needs a restart. This is how the multi-site quick start moves the first node from a public address onto a tailnet address.

How it works: the leader dials each node's new address, requires it to answer as that node, and only then commits a raft membership change that keeps the same voters at the new address. The other nodes are moved first and the leader last. If the new address is unreachable or answers as a different node, membership is left untouched. Each node serves both its old and new address for 120 seconds, so the move does not cut anyone off. The leader logs moved a member to a new rpc address for each node.

Live after the edit, with no restart: raft membership and peer dialling, the rpc listeners, the VXLAN device (recreated for the new local IP, with its forwarding entries), internal DNS, the iptables rules, node and drive addresses, relay routes, and the drive listener and NFS exports. Mounts of served drives over NFS unmount once while they are rebuilt, and drive syncs and relay routes can be briefly inconsistent during the move.

To do it:

  1. Bring the new network up on every node and check that the nodes reach each other on it, follower to follower. The leader only tests reachability from itself.
  2. Edit cluster.yaml once: the new subnet on the internal network and the new IP for every node, in the same edit. See cluster.yaml.
  3. Confirm the edit within the 60 second rollback window.
  4. Watch the leader's logs (journalctl -u clouduler -f) for one moved a member to a new rpc address line per node, then check that all nodes are healthy.
  5. If one node never moves, revert the config. The old address is still served.
Two-voter clusters

With two voters, a moved voter whose new address stops working takes quorum with it. Recover with clouduler emergency. Run three managers before re-addressing.

Quirks and gotchas​