Skip to main content

Quick start: three machines on a LAN

In quick start 1 you ran everything on one machine. Now you'll do the thing Clouduler is actually for: three machines on your home network that look after each other. At the end you'll pull the power cord out of one of them and watch your app come back on another. It takes about an hour, most of it spent installing packages three times.

You can start from scratch here, or continue from quick start 1. If you already have a single node, skip steps 2 and 3 for that machine and treat it as the first node (atlas). Everything else is identical, because adding nodes to a running cluster is exactly what step 4 does.

What you'll build​

Three managers that agree on one shared state, each holding a copy of the drive fast, with a stack that can run on any of them.

Why three?​

The managers keep the cluster's state (stacks, networks, secrets) in a small replicated database, and any change needs more than half of them to agree. That is the quorum: with three managers it is two, so one machine can disappear and the other two carry on, while with two managers losing either one stops everything. See Failover for the full story, including what happens when the lost node was running something.

Tell the page about your machines​

If you filled this in for quick start 1, your values are already here.

Your setup

What you need​

  • Three 64-bit Linux machines with systemd (Debian 13, Raspberry Pi OS on trixie, or Arch), on the same LAN subnet, with fixed addresses. They can be a mix of x86_64 and aarch64, but every node should run the same Clouduler version.
  • Root access on each.
  • SSH between your laptop and each machine.

1. Open the ports between the nodes​

Clouduler does not touch your host firewall. If you run ufw or nftables, the nodes cannot talk until you allow these. Only the LAN subnet needs the first five (the web UI listens on every interface, so the firewall is what keeps it on the LAN). Do not expose them to the internet. The ports can be changed with --rpc-port and --web-port at init; the defaults are used here.

PortProtocolUsed for
7373TCPcluster control (raft), forwarded writes, the CLI between nodes
7374TCPdrive replication
7375TCPrelay for container connections
4789UDPthe overlay network that joins stack networks across nodes
8443TCPweb UI and API, and the join command
80, 443TCPthe reverse proxy for apps

If you use ufw, on every node:

On every node (only if you run ufw)
sudo ufw allow in on {{iface}} from {{subnet}} to any port 7373:7375 proto tcp
sudo ufw allow in on {{iface}} from {{subnet}} to any port 4789 proto udp
sudo ufw allow in on {{iface}} from {{subnet}} to any port 8443 proto tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
Not set yet: {{iface}}{{subnet}} (fill in the setup form above)

These rules are an example, so adapt the interface and subnet to yours. The ports are the ones a running node listens on: raft and drive sync on 7373 to 7375 over the tailnet or LAN address, VXLAN on 4789/udp, and the dashboard on 8443.

If you never installed a firewall, there is nothing to do. See Ports and firewall for the complete list.

2. Prepare all three machines​

You repeat the preparation from quick start 1 on each machine. Here it is in one block. Run it on atlas, boreas and cirrus.

On each of the three nodes
[debian,raspbian] sudo apt update
[debian,raspbian] sudo apt install -y docker.io iproute2 iptables ethtool restic
[arch] sudo pacman -S --needed docker iproute2 iptables ethtool restic
sudo systemctl enable --now docker
sudo modprobe vxlan
sudo mkdir -p /srv/clouduler/fast

Then install the same binary on each (see quick start 1, step 2). As there, you can compile it yourself or download the prebuilt binary from the latest release. Then check it.

Copy and install, from your laptop
for h in {{ip1}} {{ip2}} {{ip3}}; do
scp clouduler-x86_64-unknown-linux-gnu {{user}}@$h:/tmp/clouduler
ssh {{user}}@$h 'sudo install -m 0755 /tmp/clouduler /usr/local/bin/clouduler && clouduler --version'
done
Not set yet: {{ip1}}{{ip2}}{{ip3}}{{user}} (fill in the setup form above)

Use the aarch64 file for any Raspberry Pi. What you should see: the same version printed three times.

Names must resolve​

The join command the web UI gives you contains the address cloud.lan.example.com:8443, not an IP. So cloud.lan.example.com must resolve to atlas on boreas and cirrus. If you have no LAN DNS, add this line to /etc/hosts on boreas, cirrus and on your laptop:

/etc/hosts
{{ip1}}   cloud.{{domain}}
Not set yet: {{ip1}}{{domain}} (fill in the setup form above)
If you see error sending request for url later

That is a node that could not resolve the name in the join command. Fix the name, or use the CLI form in step 4, which can take the IP directly.

3. Create the cluster on the first node​

If you already finished quick start 1, atlas is ready and you can skip to step 4. Otherwise run this on atlas:

On the first node
sudo clouduler init --yes \
--cluster-name {{cluster}} \
--subnet {{subnet}} \
--internal-domain {{domain}} \
--web-address cloud \
--ip {{ip1}} \
--node {{n1}} \
--drive fast=/srv/clouduler/fast
Not set yet: {{cluster}}{{subnet}}{{domain}}{{ip1}}{{n1}} (fill in the setup form above)

Every flag is explained in quick start 1, step 3. Leave --drive-mode at its default, replicated: every node keeps a full copy of the drive and containers read local disk. (The alternative, a served drive, changes how failover behaves.)

Then do the three things from quick start 1 that you must not skip: confirm sudo systemctl status clouduler is active (running), copy the first API key out of the journal (sudo journalctl -u clouduler --no-pager | grep -o 'clk_[A-Za-z0-9]*_[A-Za-z0-9]*'), and back up /etc/clouduler/secrets.key. Then sign in to https://cloud.lan.example.com:8443.

4. Add the other two nodes​

A new node needs to prove to the cluster that you allow it, and receive the cluster key without the key ever crossing the network in plain form. The web UI hides all of that behind one command.

  1. In the web UI open Nodes and click Add node.
  2. In the sheet Add a node type the Node name (boreas) and choose the role chip Manager. The sheet's default is Worker, and workers do not count towards quorum, so you must pick Manager explicitly.
  3. Fill Address with 192.168.1.11. If you leave it empty, the node detects its own.
  4. Press Create join command. The sheet changes to Run this on the new node and shows a command.
  5. Press Copy command.
/nodes
The Nodes page of a new cluster, with the Add node button.

The command looks like this (the key and pin are real in your sheet):

clouduler init --yes --join cloud.lan.example.com:8443 --token clk_... --tls-fingerprint <sha256> --node boreas --role manager --ip 192.168.1.11

It has no sudo and no --drive, so on boreas run it as sudo ... --drive fast=/srv/clouduler/fast. Then press Done on the sheet and repeat for cirrus.

What you should see on the joining node: the cluster accepted this node, installed the cluster key (...), and finally the daemon is running. Files do not need to be copied by hand: the join fetches cluster.yaml from the cluster and receives the cluster key wrapped for the new node's own key pair.

The join key is an admin key and stays valid after the join. Once both nodes are in, delete the one named join boreas (and join cirrus) under Settings, API keys and AI clients. Like every API key it has admin rights, so do not leave it around.

If you see the cluster rejected the api key

The key is wrong, was deleted, or got a stray space. Generate a new command.

If you see the cluster has no leader right now

atlas is not in a quorum yet. Check sudo systemctl status clouduler on it, and try again.

If you see ... is outside the internal subnet

The --ip of the joining node is not inside 192.168.1.0/24, the subnet chosen at init. Use an address in that subnet.

If you see the cluster already has a node called ...

A node with that name is already registered. Pick another name, or add --force if you are deliberately replacing its entry.

If you see certificate fingerprint ... does not match the pinned ...

The certificate changed between generating the command and running it, for example because a real certificate was issued meanwhile. Generate a fresh command.

If you see could not fetch the cluster key

The node joined the roster but is half set up. Copy the key from atlas over SSH and start the service:

sudo scp -p /etc/clouduler/secrets.key [email protected]:/etc/clouduler/secrets.key
# then on the joining node
sudo chmod 640 /etc/clouduler/secrets.key
sudo systemctl enable --now clouduler.service

5. Verify quorum​

sudo clouduler status

What you should see: a raft: line with quorum=yes and voters=3/3, three nodes marked up and active, a summary of the stacks, then all clear: no unhealthy instances or cluster faults. Each node line is tab-separated: name, role, reachability, availability, instance count and CPU architecture. The shape is:

cluster home
  raft: leader=atlas quorum=yes voters=3/3 state_version=... (answered by atlas)
nodes
  atlas	Manager	up	active	instances=0	x86_64
  boreas	Manager	up	active	instances=0	x86_64
  cirrus	Manager	up	active	instances=0	x86_64
stacks 0 deployed, 0/0 instances assigned to run
all clear: no unhealthy instances or cluster faults

In the web UI, Nodes shows three healthy nodes:

/nodes
Three managers, all healthy.

Because the config is replicated, sudo clouduler check on any node now reports all three nodes, and a change made in the UI reaches every one of them.

If a node shows as unreachable

Check from that node that the ports in step 1 are open, in particular 7373/tcp, and look at sudo journalctl -u clouduler -f on it.

6. Check the replicated drive​

The drive fast exists in the cluster, and each node that maps it holds a copy. Because you passed --drive fast=/srv/clouduler/fast on every node, all three do. A node can only pick a drive that already exists in the cluster, which is why you created it on atlas first.

sudo ls -la /srv/clouduler/fast

On each node, this directory should exist (it will be empty until a stack uses it). In the web UI, Storage shows the drive fast as replicated, with its resync interval (every 15 minutes by default).

Joined without --drive?

The web UI cannot add a drive mapping to a node after it has joined (a node's drives are shown read-only on its page). The drive has to be declared when the node is initialised, so re-run the join with --drive fast=/srv/clouduler/fast. A node already in the roster needs --force for that.

7. Deploy an app with a volume​

You'll reuse Vaultwarden from quick start 1. Its volume fast/vaultwarden lives on the drive you just checked, so Clouduler keeps a copy on every node.

vaultwarden.yaml
networks:
  internal: {}

services:
  vaultwarden:
    image: vaultwarden/server:latest
    restart: unless-stopped
    networks: [internal]
    volumes:
      - fast/vaultwarden:/data
    x-clouduler:
      proxy:
        - host: vault
          network: internal
          port: 80

Open Stacks, press Deploy stack, name it vaultwarden, paste the YAML, wait for Spec is valid and look at Where it can run: this time it lists all three nodes. Then press Deploy.

The scheduler picks one node. The stack page shows which. In Storage the volume appears with a Primary node (the one holding the newest copy) and Replicas, and a badge saying in sync once they agree. On a replicated drive a replica is refreshed on the drive's resync interval, so after a first deploy it may take a while for the other two to say in sync.

You reach the app the same way as in quick start 1: https://vault.lan.example.com once the name resolves to a node (any node answers, the proxy forwards to the right one).

Note which node it runs on. This page calls it boreas below; yours may differ.

8. Pull the plug​

This is the test. You're going to power off the node that runs Vaultwarden, hard, the way a real failure happens. First record the state:

sudo clouduler status

Then pull the power on that node (or sudo poweroff if you're squeamish). From another node, watch:

watch -n 2 sudo clouduler status

What you should see, with the default timers:

After the power-offWhat happens
a few secondsnothing visible yet; the node just stops answering
about 60 sthe leader decides the node is unreachable (failover.node_timeout is 60 s) and status shows it as UNREACHABLE (the UI badge reads Unreachable)
a bit laterthe scheduler places Vaultwarden on a surviving node, which pulls the image if it is not cached
a minute or so in totalclouduler stack unhealthy goes quiet and the app answers again

The timings come from the default timers, so expect them to vary a little.

In the web UI the Overview shows the node as unreachable and the stack's container with its new node:

/
A node is down; its instances have been moved.

Your cluster is still working, with two of three managers. If you now lose a second node, the quorum is gone and the cluster can no longer change anything; the remaining node is read-only until a second one returns. That is by design.

What about your data?

A replicated drive is copied on a timer (every 15 minutes by default), not on every write. If the failed node had written data in the last interval that no other node received, that data is not on the survivors. Lower Resync every on the drive (Settings, Storage) for apps where that matters, and keep backups. If a surviving node holds only an older copy, the app is refused a start while any node with the current copy is reachable. It starts from the freshest older copy only when no current copy can be reached, and then the writes since that copy are gone.

If the app does not move

Open the stack's Placement tab. It says, per node, why the stack cannot run there: a wrong pin, a missing drive on that node, not enough memory. Also remember that a stack on a public network needs a node with a public address for it. See Placement.

9. Bring it back​

Power the node on again. The service is enabled, so the daemon starts by itself and rejoins.

sudo journalctl -u clouduler -f

On the revived node you should see the daemon start and reconnect. From any node:

sudo clouduler status

What you should see: all three nodes reachable again. Vaultwarden stays where it was moved; the scheduler does not shuffle things back on its own. If you want the load spread evenly again, press Rebalance cluster on the Nodes page, or run sudo clouduler rebalance --yes. Each move restarts that instance.

Do not remove a node you want to keep

Pulling the plug is a failure. Removing a node (Nodes, Retire this node, then Remove, or clouduler node remove) is a decision, and a removed node wipes its own keys the next time it boots and refuses to rejoin. See Node lifecycle.

10. Turn on automatic updates​

Point the cluster at the official releases so it knows when a new version is out. In the UI, open Settings, then Updates, enter https://github.com/LiteApplication/clouduler as the Repository and click Save release. This can be set once from the UI. Leave Automatic checks on.

Or put it in cluster.yaml (the same keys the UI writes):

cluster.yaml
self_update:
  release_url: https://github.com/LiteApplication/clouduler
  check_updates: true
  check_interval: 24h

Clouduler then looks for a newer release every day and shows it in Settings, Updates. Nothing is installed by itself: you click Start update when you are ready, and the nodes restart one at a time. See Updates.

What happened under the hood​

  1. Every node probes its peers. When the leader's probes to the dead node went unanswered for node_timeout, it marked the node unreachable in the replicated state.
  2. The dead node, had it still been alive but cut off, would have fenced itself: after fence_timeout without quorum a node stops its own containers, so nothing runs twice.
  3. The scheduler, which runs on the leader, saw an instance whose node was unreachable and placed a new one on a node that could run it, using the replica of the volume held there.
  4. The proxy on every node routes by the shared state, so the hostname kept working through whichever node answered.

All of it is in Failover, including the emergency command for losing two of three managers.

Next steps​