Alerts
Alerts tell you when something in the cluster needs attention: a node went offline, a disk is nearly full, a container keeps crashing, a certificate is about to expire. They appear on the Alerts page and can also be sent to Discord, to any URL that accepts a JSON POST (a generic webhook), or by email.
The cluster watches about 38 kinds of event (catalog below). Each event has a severity (Critical, Warning or Info), and each destination chooses which severities it wants.
How it works
Only the raft leader (glossary) runs the alert engine. Every node can run the log watcher and the disk and memory checks, and they hand their findings to the leader. Alert settings are replicated through raft, so every node sees the same configuration. Every change to an alert is written to raft before a message is sent, so a new leader never re-sends what the old leader already sent. A newly elected leader also takes a silent snapshot first, so it does not re-announce things that already happened.
Lifecycle of one alert
- Opened. A condition holds. A new Critical alert is sent at once. A new Warning waits out the debounce window (default 5 minutes) first.
- Dropped silently. A warning that stops holding inside its debounce window is dropped and no message is ever sent. Nobody gets paged by a flapping problem.
- Escalated. A warning that becomes critical is sent again as
Escalated to <severity>: <title>(settingrenotify_on_escalation, on by default). - Resolved. When the condition has been absent for two checks in a row, the alert resolves. A "Resolved" follow-up goes to every destination that heard the opening (setting
notify_resolved, on by default). Discord can edit the original message instead of posting a new one (edit_resolved). - One-shot events. Some events are announced once and not held open: each OOM kill and each log match, plus leader changes, nodes joining or leaving, finished updates and self-update outcomes, stack deploys and removals, API key creation and revocation, secret rotation, finished volume migrations and backups that succeeded with a warning.
Severity and routing
- A Critical or Warning alert goes immediately to every enabled destination whose
severitieslist contains that severity (and whose category and stack filters match). - An Info alert is not sent one by one. Destinations with
digeston collect Info alerts into one daily digest atdigest_atin the settings time zone (default 09:00). - A per-event route overrides all of that:
muteddrops the event,in_apprecords it on the Alerts page without sending, and a list of destinations sends to exactly those destinations, ignoring their severity filters. - There are no quiet hours. The nearest tools are the digest time (Info only) and muting events.
Timing
| What | Value |
|---|---|
| Cluster conditions are re-evaluated | every 15 s on the leader |
| Disk free space check | every 60 s |
| Memory check | every 15 s |
| Warning debounce | 5 min (setting warning_debounce) |
| Alert absent before it resolves | 2 consecutive checks |
| Quorum lost message | after 60 s without quorum, once per outage |
| Delivery attempt timeout | 10 s (email: 20 s overall) |
| Delivery retries | up to 5 attempts, waiting 10 s, 30 s, 60 s, 120 s |
| Rate limit | 30 messages per destination per 10 min; extra messages are held back |
| History kept | 30 days (retention), at most 2000 entries; the Recent tab shows up to 300 |
A message held back by the rate limit is noted as held back: too many messages to this destination in 10 minutes and does not count as a delivery failure. A delivery that still fails after 5 attempts is recorded as failed for that destination and appears on the Destinations tab.
Events
Defaults can be changed per event (severity and route) on the Events tab or in the events setting. The test event used by test messages is not a real alert and is not listed.
| Event key | Category | Default | Raised when |
|---|---|---|---|
cluster.quorum_lost | Cluster | Critical | A manager has had no quorum for 60 s. Sent once per outage; the resolved message follows when quorum returns. |
cluster.emergency_mode | Cluster | Critical | The cluster is in emergency mode. |
cluster.leader_changed | Cluster | Info | The raft leader changed. Warning when it changed more than 3 times in 10 minutes ("Check the network between the managers"). |
cluster.node_joined | Cluster | Info | A node joined. |
cluster.node_removed | Cluster | Info | A node was removed. |
node.unreachable | Nodes | Critical | A node is unreachable. Warning instead of Critical for a worker. |
node.disk_low | Nodes | Warning | A drive is at or above disk_warning_percent (90). Critical at disk_critical_percent (95). Checked every 60 s. A band is entered at or above the threshold and left only when usage drops 2 points below it. |
node.memory_pressure | Nodes | Warning | Memory stays above memory_percent (92) for memory_for (10 min). |
instance.crash_loop | Workloads | Critical | A container keeps crashing. |
instance.oom_killed | Workloads | Warning | A container was killed for using too much memory. Critical on the third kill of the same instance within one hour. |
instance.exited | Workloads | Warning | A container is not in its desired state (exited or missing). |
instance.unhealthy | Workloads | Warning | A container fails its healthcheck. |
instance.unplaceable | Workloads | Warning | No node can run the container (see Placement). |
instance.unreachable_by_proxy | Workloads | Warning | The proxy cannot reach a container. |
instance.pin_drift | Workloads | Info | A container runs on another node than its pin. |
log.match | Workloads | Warning | A log rule matched a line. Severity comes from the rule. |
volume.sync_refused | Storage | Critical | A sync was refused because it would wipe data. |
volume.migration_failed | Storage | Critical | A volume migration failed. |
volume.empty | Storage | Critical | A volume that should hold data is empty. |
volume.replica_error | Storage | Warning | A replica has an error or is behind its primary for at least 30 min. |
volume.migration_completed | Storage | Info | A volume migration completed. |
restore.failed | Backups | Critical | A restore failed. Stays open for 3 days. |
backup.failed | Backups | Warning | A backup failed. Critical once failures have lasted 24 h. |
backup.overdue | Backups | Warning | No backup succeeded within twice its interval (at least 2 h). |
backup.succeeded_with_warning | Backups | Info | A backup succeeded with a warning. |
cert.expiring | Edge | Warning | A certificate expires within cert_warning_days (14). Critical within cert_critical_days (7). |
cert.issuance_failed | Edge | Warning | Certificate issuance keeps failing. |
dns.reconcile_failed | Edge | Warning | DNS records could not be updated. |
self_update.rolled_back | Deployments | Critical | A node went back to its previous binary (see Updates). |
self_update.completed | Deployments | Info | Clouduler updated on a node (sent after the new binary clears probation). |
update.failed | Deployments | Warning | An image update failed. |
update.stuck | Deployments | Warning | An image update made no progress for over 15 minutes. |
update.completed | Deployments | Info | An image update completed. |
stack.deployed | Deployments | Info | A stack was deployed ("First deploy." or "Redeployed, revision N."). |
stack.removed | Deployments | Info | A stack was removed. |
auth.api_key_created | Security | Info | An API key was created. |
auth.api_key_revoked | Security | Info | An API key was revoked. |
secret.rotated | Security | Info | A user secret was overwritten (reserved _clouduler_ secrets are not reported). |
Destinations
A destination is a named place to send alerts. Three kinds exist.
| Kind | UI label | Sends |
|---|---|---|
discord | Discord | A Discord webhook message: a rich embed or one plain line. See Discord alerts. |
webhook | Generic | A POST with a JSON body (version 1, cluster, type of alert, digest or test, the alert fields and ui_url). Headers: X-Clouduler-Event, X-Clouduler-Severity, User-Agent: clouduler/<version> and an optional Authorization. Retried on 408, 429 and 5xx; other 4xx responses are not retried. |
email | A multipart text and HTML message through the SMTP server in smtp. See Email through SMTP. |
Email subjects look like [homelab] CRITICAL: <title>, [homelab] Resolved: <title>, [homelab] Escalated to <severity>: <title>, [homelab] Daily digest: N events and [homelab] Test message from Clouduler.
Webhook and Discord URLs are stored as sealed secrets (_clouduler_alert_<name>_url, and _clouduler_alert_<name>_auth for a generic webhook's Authorization value), never in the settings. The SMTP password is _clouduler_alert_smtp_password. After saving, the UI and API show a URL only as scheme and host (https://discord.com/…).
Tasks
Add a destination and send a test message
- Web UI
- CLI / config
Open Alerts (/alerts), then the Destinations tab, and click Add destination (admins only). Choose a type chip: Discord, Generic or Email. Enter a Name (it cannot be changed later), the Webhook URL (or URL, or Recipients for email), choose Send these severities (Critical, Warning, Info, as a daily digest), and click Send test message.
For email, set up the server first in the SMTP panel on the same tab: Host, Port, Security, Username, Password and From, then use Send a test email to.
Each destination shows a status: Disabled, Failing, Working or Not used yet. A banner reads "NAME has failed N deliveries in a row" when sends keep failing.
There is no CLI command for alerts. Use the web UI or the API (GET/PUT /api/alerts/settings, POST /api/alerts/test, see the HTTP API). Only admins can save settings or send tests.
curl -X POST https://cloud.example.com/api/alerts/test \
-H "Authorization: Bearer $CLOUDULER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "ops-discord"}'
How a test works. A test sends one message with a 10 second limit through the saved destination, or through what you typed if you changed it. Stored credentials only ever go to the address they were saved for: a stored Authorization value is used only when the test URL equals the stored URL, and the stored SMTP password only when host, port and username match the stored server. If you change the URL while testing, enter the Authorization value again.
Route one event differently
- Web UI
- CLI / config
On the Events tab, open an event. Set Severity (Default, Critical, Warning, Info) and Goes to (Default, In-app only, Pick destinations, Muted). Reset to default undoes it.
Settings live in the replicated config _clouduler_alerts. As stored (JSON):
{
"events": {
"cluster.leader_changed": { "severity": "warning" },
"node.disk_low": { "route": { "destinations": ["ops-discord"] } },
"auth.api_key_created": { "route": "in_app" }
}
}
Change timing and thresholds
- Web UI
- CLI / config
On the Alerts page, the timing panel has Warning debounce, Daily digest at, Time zone, Keep history for and Public URL, the toggles Send notifications, Notify again when an alert gets worse, Tell me when it is resolved and Display warnings during debounce, and a Thresholds panel for disk, memory and certificate numbers. Changes apply on the next pass, no restart needed.
Edit the same keys through PUT /api/alerts/settings. See Options.
Read the Recent tab
The Recent tab (/alerts/recent) shows the metrics Critical now, Warnings now and Destinations, severity filter chips, Active now (worst first) and Earlier (grouped as Today, Yesterday, Earlier this week, Earlier). Row badges: Critical, Warning, Info, Resolved, Waiting (a warning still inside its debounce), Was followed by the old severity (escalated) and New (since you last pressed Mark all as read, which is stored in your browser). Show N log lines expands a log excerpt.
Log watcher
Add logs rules to a stack's x-clouduler.alerts block to raise an alert when a line appears in a container's output. The watcher runs on every node, follows only containers that have rules, and starts at the end of the log (docker logs --follow from zero lines), so lines written before it started are never matched.
x-clouduler:
alerts:
enabled: true
mute:
- instance.oom_killed
- backup.*
severity:
instance.exited: critical
logs:
- match: "database is shutting down"
severity: critical
title: "Database shutting down"
cooldown: 10m
stream: both
- regex: "panic|FATAL"
ignore_case: true
severity: warning
services: [app]
services:
app:
image: example/app:1.0
x-clouduler:
alerts:
mute: [log.match]
- A rule sets exactly one of
match(literal text) orregex. - Cooldown. After a rule fires for a container it stays quiet for
cooldown(default 5 minutes). Matches inside the cooldown are counted, and the next message saysIt matched N more times in the last …. - Excerpt. The watcher waits 1 second after a match so nearby lines join the excerpt. It keeps at most 3 lines, each cut to 500 characters. Lines over 8 KB are cut.
- Caps. At most 30 log signals per minute per node; the queue holds 64. A flood of matches can silently lose some.
- Rules are only accepted in the stack-level block. A service-level block can set
enabled,muteandseverity; it is merged over the stack block. muteaccepts event keys or aprefix.*pattern that covers a whole category.enabled: falsemutes every event for the stack, including its log rules.- The container is only watched when its stack spec contains the text
logs.
Options
Alert settings (_clouduler_alerts)
All settings apply live.
durationdefault 5mHH:MMdefault 09:00IANA nameno defaultbooldefault truebooldefault falsedurationdefault 30dURLno defaultobject or nullno defaultmapdefault {}Destination (each item of destinations)
namerequiredstringno defaultkindrequiredwebhook | discord | emailno defaultsetdefault critical, warninglistdefault allURL (secret)requiredURLno defaultstringno defaultstringno defaultping | normal | silentdefault pingping | normal | silentdefault normalping | normal | silentdefault silentping | normal | silentdefault silentbooldefault truebooldefault falsetorequiredlist of addressesno defaultSMTP server (smtp)
integerdefault 587stringno defaultRouting override (events.<key>) and thresholds
info | warning | criticaldefault catalog defaultmuted | in_app | {destinations: [...]}no defaultinteger 0-100default 90integer 0-100default 95integer 0-100default 92integerdefault 14integerdefault 7Stack alerts (x-clouduler.alerts)
Take effect when the stack is deployed. x-clouduler.alerts: false turns all alerts off for the stack.
listdefault emptymapdefault emptylist of rulesdefault emptylogs[].match / regexrequiredstringno defaultdurationdefault 5m