Skip to main content

Git backup

Git backup mirrors the cluster's configuration into a Git repository you own. If every node is lost, you can rebuild the cluster from that repository. It also lets you edit stacks in Git and have the cluster pick the change up.

It backs up configuration, not your data. For drive data, see Backups.

What is exported​

The leader pushes after every change, to the branch you configure:

Path in the repositoryContent
cluster.yamlThe cluster config, plaintext.
stacks/<name>.yamlEach stack spec, plaintext.
configs/<name>Each config, plaintext.
secrets/<name>.ageEach secret, encrypted with age to your recipient key.
.clouduler-secret-revisions.jsonBookkeeping.

Only the cluster config, stacks, configs and secrets are exported. Backup destinations and schedules are not. Export runs on the leader only and pulls first; a commit that fails validation is parked on a branch named rejected/<branch>-<sha12>. The deploy key is a reserved secret named _clouduler_git_deploy_key.

Nothing is committed when nothing changed. Files in the repository that Clouduler does not manage are kept. Keep the repository private: configs and stack files are readable by anyone with access.

Secrets are encrypted to a key whose private half you keep. Clouduler never stores the age identity, so without it the secret values in Git cannot be recovered (stack and config files still can).

Set it up​

Open Settings, Sign-in and backup (/settings/security), page Git backup.

/settings/security?page=git-backup
Settings, Sign-in and backup, Git backup.
  1. Tick Back up cluster state to git.
  2. Repository URL: SSH form, for example [email protected]:ops/clouduler-state.git. HTTP(S) is rejected.
  3. Encrypt secrets for: paste an age public key (age1...), or click Generate age key. The browser creates a key pair and downloads clouduler-age-key.txt (the private key never leaves your browser). Store that file safely, outside the cluster.
  4. Branch (default main).
  5. Save. A deploy key public key appears; click Copy public key and add it to the repository as a deploy key with write access.
  6. Wait for the first export. Last export shows success or Last export failed: ....

The deploy key is generated by the leader on the first export and stored as a reserved secret, so the first push fails until you add it to the repository.

Webhooks and pull​

Git is also an input. When you push a change to the branch, the cluster can pull it, validate it and apply it. The pull is triggered by a webhook, or by hand.

On the same page, set a webhook token and click Save webhook token. Configure your Git host to POST to /api/git-backup/webhook/<token>. The UI shows the URL pattern. Make sure that address is reachable from the Git host; see the git webhooks guide.

The UI has no buttons for export now or pull now.

The webhook accepts any request with the right token (no event filtering, no signature). A wrong token returns 401 bad webhook token. The token is in the URL, so it can end up in the access logs of any reverse proxy in front of the dashboard.

The webhook is part of the dashboard API, so it is reachable wherever the dashboard is. With the default web.through_proxy: true, that is the dashboard hostname on the web.network you chose, plus the direct dashboard port (8443 by default) on every address of the node. A Git host on the internet can only reach it if the dashboard is on a public network or that port is forwarded to it. The daemon's own log does not record request URLs at the default log level.

What a pull does​

  • It applies a changed cluster.yaml (through the same generation check as other config edits, so a stale pull is rejected), redeploys only stacks whose spec changed, removes stacks no longer in Git, and updates or removes non-reserved configs.
  • It validates first and rejects the commit otherwise, moving it to a branch named rejected/<branch>-<sha12>:
    • no cluster.yaml in the pulled tree
    • the pulled tree has no stacks/ at all, which would remove all N stack(s) this cluster runs - remove stacks through the api, not by emptying the repo
    • stacks/NAME.yaml: config C has no content under configs/
    • secret S does not already exist in the cluster - git cannot supply new secret content, create it through the api first
    • a commit that changes anything under secrets/: ... secret rotation must go through the api, not git
  • Secret values never come from Git. Rotate them in the UI or with clouduler secret set.

Export is leader-only and runs a pull first, so it does not overwrite a manual push. A new leader builds its first export on top of the remote branch, not on an empty history. Older versions had races here that redeployed every stack on each pull and reverted manual pushes; they are fixed.

Disaster recovery​

If the cluster's content is gone from every node, rebuild it from the repository. Clouduler's own deploy key is stored in the lost state, so clone the repository yourself first.

git clone [email protected]:ops/clouduler-state.git ~/clouduler-state

sudo systemctl stop clouduler.service       # the daemon must be stopped
sudo clouduler restore --age-identity-file ~/clouduler-age-key.txt \
  --branch main --yes ~/clouduler-state
sudo systemctl start clouduler.service

restore takes the repository path (or an ssh:///git@ URL, cloned with your SSH agent). Use --age-identity-stdin instead of the file to pipe the identity in. The identity is read once, used in memory, and each secret is re-sealed under this node's cluster key. Secrets that cannot be decrypted are listed as skipped. restore refills content only; it does not repair quorum. If the cluster also lost its managers, run emergency mode first. Then restore your data from Backups.

Options​

Option
Type / default
Reload
Description
backup.repo_urlrequired
stringno default
live reload
SSH URL of the repository.
Watch out: Errors: backup.repo_url must not be empty; only ssh urls are supported (e.g. git@host:org/repo.git), not http(s).
backup.age_recipientrequired
age public keyno default
live reload
Recipient the secrets are encrypted to.
Watch out: Error: must be an age X25519 recipient, starting with age1.
backup.branch
stringdefault main
live reload
Branch to push to and pull from.
Watch out: Must not be empty.
backup.interval
stringno default
n/a
Accepted and ignored. Exports now happen on every change.
restore --age-identity-file / --age-identity-stdinrequired
path / flagno default
n/a
Age identity that decrypts the secrets. One of the two.
restore --branch
stringdefault main
n/a
Branch to read when the source is a URL.
restore --node
node namedefault hostname
n/a
This node name. Also CLOUDULER_NODE.
restore --yes
flagdefault off
n/a
Skip the confirmation prompt.
git-backup pull|reconcile --via
node namedefault any manager
n/a
Node to ask. A non-leader forwards to the leader.

Quirks and gotchas​