Threshold alerts for host metrics in ClickStack: every host, or one server group

Two alerts an infra team asks for first, built in the ClickStack chart builder:

1. One threshold for every server — for example memory used at least 90%, with a separate notification for each host that crosses it.
2. A threshold for one server group only — the hosts whose svc.group resource attribute (set from an EC2 tag, see labs/common-fields) is game-a.

Verified on Managed ClickStack, our ClickHouse Cloud Seoul service (ClickHouse 26.8.1.654), 2026-11-11 04:57–05:08Z, one run. managed/threshold_alerts.py created the metric source, dashboard, builder charts, alerts and a generic webhook through the ClickHouse Cloud API, sent synthetic system.memory.utilization for six hosts every 20 s, and counted the notifications at a local receiver behind a tunnel. Screens are from the same run. Report: results/2026-11-11-managed.md.

Result

Six hosts, two groups (game-a-01..03, game-b-01..03), all at 0.50; for six minutes game-a-01 at 0.95, game-a-02 at 0.85, game-b-02 at 0.93; then all back at 0.50.

Alert (builder line chart, Average, group by host.name, every 1 m)	Notified	Not notified	First window	Resolved
A1 every host, ≥ 0.9	game-a-01, game-b-02 (6 each)	game-a-02 (0.85) and the rest	05:00–05:01	05:06–05:07, per host
A2 svc.group = 'game-a' only, ≥ 0.8	game-a-01, game-a-02 (6 each)	game-b-02 (0.93, other group)	05:00–05:01	05:06–05:07, per host
A3 as A1 with 3 consecutive windows	game-a-01, game-b-02 (5 each)	as A1	05:02–05:03 (2 windows Pending)	05:07–05:08, per host

What this means:

1. One rule covers every host. Group the chart by host.name; the alert then evaluates each host as its own group and notifies (and later resolves) each host on its own. The alert's history shows the groups of every window (screen 7).
2. A server group is a filter on the same chart. Add the group attribute to the chart's Where. A different threshold per group is one chart and alert per group.
3. A breaching host is notified every window (6 notifications in a 6-minute breach). Consecutive windows delays the first notification (here 2 windows Pending, then Alert) and still repeats after that; with many hosts, also look at the interval and how the receiving channel groups messages.

Step by step

1. A chart grouped by host. Dashboards → Add → Time Series. Data Source: your metric source. Average of system.memory.utilization, Where (SQL) Attributes['state'] = 'used', Group By ResourceAttributes['host.name']. The hostmetrics memory scraper sends one series per state (used, free); the filter keeps one series per host.

2. The alert on that chart. Edit alert → Static threshold, "is at least (≥)" 0.9, Send to a webhook (Slack or generic). The message template may use {{group}} and {{value}}.

The panel showed "over 5 minute" for these alerts, created with interval: 1m through the API; the alert's own page says "Evaluates every 1m", and notifications came every minute. Pick the interval you want when you create the alert in the UI.

3. One server group. The same chart with the group in Where: Attributes['state'] = 'used' AND ResourceAttributes['svc.group'] = 'game-a', alert at 0.8.

4. Fewer notifications for short spikes. Alert → Advanced Settings → Consecutive windows.

5. Where to see it. The dashboard (the bell turns red while an alert fires), the Alerts page, and each alert's page with one row per window and, expanded, one row per host.

The alert pages show local time (KST here); the notifications and the report use UTC.

What to know before rolling it out

• A gauge is read as the last value in each window. "Average" on a gauge metric takes each series' last value in the window, then averages across series (last_value(Value) in packages/common-utils/src/core/renderChartConfig.ts, HyperDX source read 2026-11-11). Grouped by host with one series per host, that is the host's latest sample. Seen here: the breach started at 05:00:51 and the 05:00–05:01 window already fired. With the hostmetrics default of one sample per minute, each window holds one sample anyway.
• The notification title rounds the value to one decimal: 0.93 arrived as "0.9 meets or exceeds 0.9". {{value}} in the message carried 0.93. "is at least" is ≥.
• The group label is the expression, as received: Group: "arrayElement(ResourceAttributes, 'host.name'):game-b-02".
• A host that stops sending does not fire a threshold alert — there is no value to compare. That needs the expected-host-list alert in labs/process-down-alerts.
• Consecutive windows, as measured once: A3 fired from its third window and resolved one window after A1 (05:07–05:08 against 05:06–05:07), on the same chart query.

Run it

ngrok http 9310                    # or any tunnel to 127.0.0.1:9310; optional
managed/threshold_alerts.py --webhook-url https://<tunnel> [--keep]   # ~14 min
managed/threshold_alerts.py --cleanup                                  # after --keep
managed/threshold_alerts.py --report <date>                            # report again from results/

Env files: CH_HOST, CH_USER, CH_PASSWORD (--ch-env) and CHC_KEY, CHC_SECRET (--api-env). Everything it creates is named threshold-alerts lab and lives in the database threshold_alerts_lab; --cleanup removes it. Without --webhook-url the webhook points at example.com and only alert states are recorded (the Cloud API reports one state per alert, not per host). Links and per-alert hashes are redacted before anything is written to results/.

File	What
managed/threshold_alerts.py	source, dashboard, alerts, webhook; data sender; receiver; report
results/2026-11-11-managed.md	alert states every 30 s and every notification
results/2026-11-11-webhook.jsonl, -managed-states.jsonl, -phases.json	the raw records
images/	the screens above, from the same run

Not covered: a Slack channel (the generic webhook carries the same text), CPU (system.cpu.utilization has one series per CPU and state, so its filter and threshold differ — not run here), one rule with a different threshold per group, silences, real hosts (the data is synthetic in the hostmetrics shape), and the local OSS build.

---

ngrok http 9310                    # 127.0.0.1:9310으로 가는 터널이면 무엇이든; 선택
managed/threshold_alerts.py --webhook-url https://<tunnel> [--keep]   # 약 14분
managed/threshold_alerts.py --cleanup                                  # --keep 뒤에
managed/threshold_alerts.py --report <date>                            # results/에서 보고서 다시 만들기