Skip to main content

Backups

Clouduler backs up drives with restic, an encrypted, deduplicating backup tool. You define a destination (where the backup repository lives), then a schedule (a drive, a destination, how often, how long to keep snapshots). The cluster runs the backups on its own, keeps a history, and can restore a whole drive, one volume, or everything a service uses.

Drive backups are separate from git backup, which saves the cluster's configuration and secrets. You want both: git backup rebuilds the cluster, restic brings your data back.

restic must be installed

Nodes that run backups need restic on their PATH. If at least one schedule is enabled and it is missing, the daemon logs restic is not on PATH but at least one backup target is enabled - scheduled backups will fail until it is installed.

How it works​

  • A destination is a restic repository: a folder on a cluster node, an SSH server, or an S3 bucket.
  • A schedule (called a target in the CLI) ties one drive to one destination. Its id is DRIVE-DESTINATION, for example fast-nas.
  • The leader checks for due schedules every 30 seconds and queues a run. The node that holds the drive runs restic: it optionally stops the containers using the drive, backs up, restarts them, then applies retention (forget --prune). Failures in retention are logged as warnings and do not fail the run. A run waits up to 120 seconds for the drive to finish syncing first.
  • A schedule is due when the frequency has elapsed and, if you set a time of day (UTC), the clock has reached it.
  • The run history keeps the last 50 runs per schedule.
  • A restore request is planned by the cluster and executed on the node that is the drive's authority, so you can ask from any node.

The restic password​

Every destination has a repository password, generated automatically by the CLI (64 hex characters) and sealed in the cluster. It is never shown. It is recoverable only through the cluster key or from a git backup (age-encrypted, reserved secret _clouduler_restic_password_<destination>). If you lose the cluster key and the age identity, every backup in that repository is unreadable. Keep both safe.

Add a destination​

Open Storage, tab Backups (/storage/backups). With none configured, a banner says "No destinations yet. Add one under Destinations, then schedule a drive to back up to it." Under Destinations click New destination and choose a kind:

  • Cluster node: pick the Node and a Path (default /srv/backups, no spaces). If the node does not hold the drive, the drive is synced there once before each backup.
  • SSH server: Host, Port, User, Path and the Private key (sealed, never shown again).
  • S3 bucket: Bucket, Endpoint, Region, Prefix, Access key ID, Secret access key.

Name it, then click Create destination. The form says to leave Repository password blank to generate one, but the server currently rejects a blank password for a new destination with password is required for a new destination. If you hit that, create the destination with the CLI.

/storage/backups
Backups: schedules, recent runs, destinations, restores and snapshots.

Schedule a backup​

Under Schedules click New schedule (disabled until there is a drive and a destination). Fields: Drive, Destination, Frequency (hours) (default 24), Time of day (UTC) (optional), Keep last N snapshots (default 7), Keep snapshots within (days), Stop containers before backup (Use the stack's setting (default: yes), Always stop, Never stop) and Enabled (default on). Click Create schedule.

Use Run NAME now on the row to test, then check Recent runs. A failed latest run shows a banner, "The last fast backup failed".

Containers that use the drive are stopped during the backup by default so the files are consistent, then restarted. A stack can set its own preference; the schedule can force it either way. For databases, consider a dump job in addition.

History​

Recent runs shows the latest 20: drive, destination, start time, who started it (schedule or manual (user)), and the outcome (snapshot short id, size, any warning, or failed). The Schedules table shows each schedule's last run and a paused or enabled tag. The drive page (/storage/drives/fast) lists that drive's schedules too.

A run that stops reporting is marked failed only after three times the frequency (minimum 12 hours, maximum 72 hours), so a dead run can look "Running" for a long time. Failed runs are shown as outcomes, and they also raise the backup.failed alert (see Alerts). The UI shows only the 20 most recent runs, and each target keeps at most 50.

Restore drill​

Restoring overlays the snapshot onto the live data. It stops every container that uses the data, overwrites the files, replaces the copy on every node, then starts the containers again. Files that exist now but not in the snapshot are kept, not deleted. Rehearse this on something you can afford to lose before you need it.

  1. Check the backup exists. In Storage, Backups, scroll to Snapshots, choose the destination chip and find a recent snapshot. On the CLI, clouduler backup snapshots nas (this may fail for S3 and key-based SSH destinations because the CLI does not pass those credentials to restic; use the UI there).

  2. Pick the scope.

    • Whole drive: every volume on the drive.
    • Single volume: one drive/volume.
    • A service's volumes: every volume a service mounts; the service restarts once.

  3. Start the restore.

    Click Restore from backup (on Backups, or the drive page). Choose the scope, Drive (and Volume, or Stack and Service), Destination (Resolve automatically for services with one destination) and Snapshot (Latest or a dated one). The sheet warns "Restoring overwrites what is there now." Type the target name, then click Restore fast. A toast says "Restoring fast. Follow it under Restores."

    For a single volume, the UI currently sends drive/volume where the server expects only the volume name, so it fails or targets the wrong volume. Use the CLI for single-volume restores.

  4. Watch it. The Restores section lists the request with its status (Failed: ERROR on failure).

  5. Verify. Check the containers are running (clouduler stack unhealthy --all), open the app, and look at the data.

  6. If it fails. Midway failures leave containers stopped on purpose: restoring fast failed; N container(s) were left STOPPED because fast may be half-restored - check the data, re-run the restore, then start them again. Fix the cause, re-run the restore, then start the stack. If it fails before changing anything: restore aborted before changing any data; containers were restarted.

Restoring onto a replica is refused, since the next sync would overwrite it: drive X is served by Y; restoring must run there or ... is not the primary of ID (the primary is NODE) .... Requests sent through the web UI are handed to the right node automatically.

Restoring after losing the whole cluster​

  1. Install a new cluster and restore its configuration from git backup; that also brings back the sealed restic passwords, re-sealed under the new key.
  2. Make sure the backup destinations exist again. Git backup does not export destination or schedule records, so you re-add them by hand.
  3. Add each destination with the same id and run the restore commands above.

Options​

Option
Type / default
Reload
Description
destination add ID --kindrequired
node-local | ssh | s3no default
n/a
Destination type.
Watch out: node-local needs --node and --path; ssh needs --host, --user, --path; s3 needs --bucket, --endpoint.
--namerequired
stringno default
n/a
Human-readable name.
--port
intdefault 22
n/a
SSH port.
--region / --prefix
stringno default
n/a
S3 region and key prefix.
--password
stringdefault generated
n/a
Repository password. Omit to generate and seal one.
Watch out: Re-adding an existing id without it rotates the password and locks you out of the repository.
--ssh-key-file
pathno default
n/a
SSH private key sealed as the credential.
Watch out: Conflicts with the S3 key flags.
--s3-access-key-id / --s3-secret-access-key
stringno default
n/a
S3 credentials, sealed. Must be given together.
target set ID --driverequired
driveno default
n/a
Drive to back up.
--destinationrequired
destination idno default
n/a
Destination to back up to.
Watch out: Only one enabled schedule per drive and destination.
--frequencyrequired
durationno default
n/a
How often to run, e.g. 12h, 1d. Minimum 1 second.
--time-of-day
HH:MM (UTC)default any time
n/a
Earliest time of day to run once due.
Watch out: A malformed value is rejected by the CLI.
--keep-last
intdefault no limit
n/a
Keep at least this many recent snapshots.
--keep-within
durationdefault no limit
n/a
Keep every snapshot within this age, e.g. 30d.
Watch out: Do not pass 0s (becomes 1h).
--stop-containers
booldefault stack setting, then true
n/a
Stop containers using the drive during the backup.
--enabled
flagdefault off
n/a
Enable the schedule. Without it, it is created disabled.
Watch out: The UI form defaults to enabled.
restore-* --restore-from
destination idno default
n/a
Destination to restore from. alias --destination.
Watch out: Optional for restore-service when only one exists. Errors: drive X has no backup target on record; pass --restore-from explicitly / drive X backs up to N destinations (...); pass --restore-from to pick one.
restore-* --snapshot
latest | iddefault latest
n/a
Snapshot to restore.
Watch out: Errors: snapshot ID is not a snapshot of drive X in this repository; this repository has no snapshots of drive X yet; snapshot id ID is ambiguous; give more characters.
restore-* --yes
flagdefault off
n/a
Skip the confirmation prompt.
all backup subcommands --via
node nameno default
n/a
Send the request through this node.

Quirks and gotchas​