Process-down alerts in ClickStack: does an alert notice a series that stopped?

The usual OpenTelemetry way to watch "is the server process up" is the hostmetrics process scraper (or windowsperfcounters) filtered to one executable. When the process stops, its series stops — nothing is sent for a process that is not there. This lab checks what five ClickStack alerts do about that, on a local ClickStack all-in-one 2.42.0 (the HyperDX version Managed ClickStack runs), with two hosts and one of them losing its process.

Verified on ClickStack all-in-one 2.42.0 (ClickHouse 26.8.34.13), 2026-11-06, one run of the scenario below. Notifications counted at the lab's webhook receiver. Full report: results/2026-11-06-run.md.

Result

Alert (all below 1, every 1 min)	One host's process stops	Every host down	Back up
T1 builder line, count, group by host.name	silent	fires, as one ungrouped alert	state OK, no resolve notification
T2 SQL line, points per host	silent	fires, ungrouped	state OK, no resolve notification
T3 SQL line against the hosts seen in the last hour (0 where a host sent nothing)	fires for host:host-b after 76 s	fires per host	resolve per host
T4 builder number tile filtered to one host	fires after 76 s	fires	resolve
T5 builder line, count, no group	silent	fires	resolve

What this means:

1. A grouped alert does not notice one missing group. HyperDX evaluates the rows a bucket returns; a host with no rows is not a row. Read in packages/api/src/tasks/checkAlerts/index.ts at @hyperdx/api@2.42.0 (time-series path) and now seen in a run. A bucket with no rows at all is evaluated as 0, which is why every alert fires when every host stops.
2. The collection method decides it. A method that stops sending (hostmetrics process, windowsperfcounters) needs an alert that supplies the expected list itself (T3). A method that sends an explicit "stopped" value (windows_service, a check script that sends 0) works with the plain grouped alert.
3. T3 is a SQL tile alert, and Managed ClickStack accepts alerts on SQL tiles (used in labs/apm-workflows). Here the expected list is "hosts seen in the hour before the window"; an inventory table or EC2 tags would be the production version (a host retired on purpose then stops alerting after an hour).
4. A breaching alert notifies every minute (5 notifications in a 5-minute stop). With thousands of hosts that needs a longer interval, numConsecutiveWindows, or routing that de-duplicates.
5. Time to first notification: 76 s from the stop with 20 s sends and a 1-minute alert. With the scraper's 60 s interval, add up to one more interval.

The same test on Managed ClickStack

Verified on Managed ClickStack, our ClickHouse Cloud Seoul service (ClickHouse 26.8.1.608), 2026-11-06 06:49–07:08Z, one run. managed/managed_alerts.py creates the dashboard, a generic webhook and the alerts through the ClickHouse Cloud API (SQL tiles only), inserts the two hosts' rows over HTTPS every 20 s, reads each alert's state every 30–40 s, and deletes everything at the end. The webhook pointed at example.com, so nothing was delivered; executionErrors stayed empty. Report: results/2026-11-06-managed.md.

Alert (SQL tile, below 1, 1 m)	host-b stopped	every host down	back up
M2 points per host, group by host	OK throughout (silent)	ALERT	OK
M3 against the hosts seen in the last hour	ALERT within about 2 min	ALERT	OK
M4 number tile, host-b	ALERT within about 2 min	ALERT	OK
M5 all points, no group	OK	ALERT	OK

Same as the local run: a group-by alert does not fire for a host whose series stopped; the expected-list SQL tile and a per-host number tile do. The 2 minutes are bounded by the 30–40 s state polling, not measured to the second (locally the first notification came at 76 s).

Log alerts on Managed ClickStack (managed/managed_log_alerts.py, results/2026-11-06-managed-logs.md, 2026-11-06 08:06–08:22Z): two services logging every 20 s; svc-b throws errors for 4 minutes, then sends nothing for 5.

Alert (SQL tile, 1 m)	svc-b error burst	svc-b silent
E1 errors per service, at least 5	ALERT	OK
E2 log lines per service, below 1	OK	OK throughout (silent)
E3 log lines vs the services seen in the last hour, below 1	OK	ALERT

A condition that produces rows (errors) works as a group-by alert; a service that stopped logging needs the expected list, as for metrics. A second metric run with --keep (screens captured, then --cleanup) gave the same table as the first: results/2026-11-06-managed-run2.md. On screen, a chart draws a bucket with no data as 0 — the M2 chart looks as if host-b fell to 0 — while the alert does not evaluate that group; and an alert created with interval: 1m through the Cloud API showed "over 5 minute" in its settings while being evaluated every minute.

Run it

cp .env.example .env               # ports 29xxx, clear of _base/ and labs/promql
docker compose up -d               # ClickStack 2.42.0 + webhook receiver
bin/setup.py                       # local test user (.env), webhook, dashboard, 5 alerts
bin/run.py                         # 23 minutes: P0 both up 5 · P1 host-b stopped 6 · P2 both up 4 · P3 all down 5 · P4 both up 3
bin/summarize.py > results/$(date -u +%F)-run.md
docker compose down

File	What
bin/sender.py	stand-in for the process scraper: process.memory.usage for GameServer.exe, per host in out/hosts, every 20 s (so a 1-minute bucket is never empty by jitter)
sql/t2_grouped.sql, sql/t3_expected.sql	the two SQL tiles; T3 crosses the window's buckets with the expected hosts and left-joins the counts
bin/webhook.py	the receiver: every POST to out/webhook.jsonl with its arrival time
bin/run.py, bin/summarize.py	the phases, alert state every 30 s from the API, the report

Not covered: Slack as the channel (the generic webhook carries the same text), silences or maintenance windows, and the Cloud API path for creating the same alerts (that is labs/apm-workflows). The "end" rows in the report are the sender stopping at teardown, not part of the scenario.

---