Vault audit trail and HA metrics in ClickStack

HashiCorp Vault writes an audit entry for every request and response: who logged in, which token or certificate was issued, which secret or key was used, what was denied. This lab sends that trail from a three-node Vault OSS cluster (Raft integrated storage, one active node, two standbys) through an OpenTelemetry Collector into ClickHouse, shaped as OTel logs that ClickStack reads as they are. The same Collector scrapes each node's own metrics, so the active/standby state sits next to the audit trail.

Verified on hashicorp/vault 2.2.1 (OSS, 3-node Raft), otelcol-contrib 0.160.0 and clickstack-all-in-one 2.42.0 (ClickHouse 26.8.34.13), locally, 2026-11-06: bin/check.py 20/20 PASS, including a failover of the active node (results). Managed ClickStack: the same Collector configuration with CH_* pointed at a ClickHouse Cloud service (26.8.1.608, ap-northeast-2) and Managed ClickStack HyperDX 2.42.0-sha6e4d740, 2026-11-06: 19 PASS, 1 SKIP (S7's negative runs locally only) (results); the two sources and the dashboard set up in the UI as below, all seven tiles drawn.

What it answers

Question	Answer in this lab
Can Vault's access and issuance history go to ClickHouse?	Yes. The file audit device writes one JSON line per request and per response; the Collector's file_log receiver reads it, one row per line (S1, S6, S7)
Can it look like OpenTelemetry?	Yes, in the Collector: severity, EventName, the entry's own timestamp, TraceId from Vault's request id (request and response share it), flat vault.* / client.* attributes, service.name=vault (S1, S4). Vault itself emits no OTLP
Is it usable for audit?	Secrets, tokens and accessors stay HMAC'd as Vault wrote them, and no value appears in clear (S2). Issuance joins to later use on the HMAC'd accessor; sys/audit-hash turns a known token or accessor into the value to search for (S3)
Active/standby?	Every node writes the entries it handles as active to its own file; standbys forward requests and write nothing. Collect from every node. After a failover the new active node's file carries on, nothing is lost (S6)
Metrics per node?	Only with unauthenticated_metrics_access on the listener: without it a standby's /v1/sys/metrics answers 307 to the active node. With it, vault_core_active is 1 on exactly one node (S6, S8)
Managed ClickStack?	The Collector writes straight to the Cloud service with the ClickHouse exporter; only CH_* changes. Sources are created in the HyperDX UI (below)

What it builds

 vault-1 ─┐ audit.log (own volume)        ┌─ file_log ─ transform (OTTL) ─┬─ clickhouse ─▶ vault_audit.otel_logs
 vault-2 ─┼──────────────────────────────▶│                                └─ count ──┐
 vault-3 ─┘ /v1/sys/metrics (per node)    └─ prometheus ───────────────────────────────┴▶ vault_audit.otel_metrics_*
                                         otelcol (one here; one per Vault host in production)

Piece	Where	What
Vault OSS 2.2.1 ×3	docker-compose.yml, vault/config.hcl	Raft integrated storage with retry_join, TLS off (lab only), telemetry on, unauthenticated_metrics_access on the listener
Init and unseal	bin/vault_ha.sh	one Shamir share (lab shape); the unseal key and root token go to .run/vault-init.json, mode 600, never printed
Audit device and engines	bin/setup_vault.sh	file device mode=0644; KV v2, transit key orders, pki/ (serial number left out of HMAC) and pki-untuned/; policies app-ro, ops-finance; AppRole orders-api, userpass
Traffic	bin/traffic.sh [rounds] [node]	per round: an AppRole login, KV reads, transit encrypt/decrypt, two certificates, two denied requests, a userpass login, a token created by an administrator
Collector	otelcol/config.yaml	file_log + json_parser (timestamp from the entry), transform to OTel fields, count connector, prometheus on three targets with a keep list, file_storage for read offsets
ClickStack 2.42.0	docker-compose.yml, bin/setup_hyperdx.py	local HyperDX with "Vault Audit" and "Vault Metrics" sources on vault_audit, and the dashboard below
Dashboards	dashboard/vault-audit.json, dashboard/vault-audit.ko.json	"Vault audit and HA" and its Korean twin "Vault 감사와 HA" (same tiles, Korean names and column aliases): active node, scrape up and audit entries per node; responses by category, denied requests, issuance, secret and key use. Both import as they are on Managed ClickStack (Dashboards → Import)
Screens	bin/capture.mjs	headless Chromium, HyperDX dark mode, 1-minute granularity: dashboard, denied requests, one request/response pair, a login's detail panel, into images/ko/ and images/en/
Questions	sql/queries.sql	issuance history, one token's whole life, denied requests, secret and key use, HA state now, which node held the trail, responses per category

The OTel shape

OTel field	From the audit entry
Timestamp	time
TraceId	request.id without hyphens: request and response rows share it
EventName	vault.audit.request / vault.audit.response
SeverityText	INFO; WARN when the policy denied it; ERROR for any other error
Body	the original JSON line, unchanged
ResourceAttributes	service.name=vault, deployment.environment.name
LogAttributes	vault.node, vault.request.{id,operation,path}, vault.mount.{type,point}, vault.namespace, client.address, client.port, user_agent.original, vault.auth.{display_name,entity_id,policies,token_type,accessor,client_token,role_name,username}, vault.policy.allowed, vault.error, vault.issued.{accessor,policies,ttl_s,entity_id}, vault.lease.{id,ttl_s}, vault.pki.{serial_number,expiration}, vault.transit.key_version, vault.event.category

vault.event.category is the Collector's reading of the path: token_issue, cert_issue, credential_issue (<mount>/creds/, <mount>/sts/), secret_read, secret_write, key_use, denied, admin (sys/), preflight, other.

Findings (measured 2026-11-06, Vault 2.2.1 OSS)

• Audit is per node. A request sent to a standby was written only to the active node's audit file; the standbys had no audit file at all until they became active. After the active node was stopped, a standby took over in about one second and the next entries were in its file. The Collector therefore reads every node's file — one agent per Vault host in production.
• Standby metrics need unauthenticated_metrics_access. Without it, /v1/sys/metrics on a standby answered 307 to the active node even with a valid token, so a scrape of a standby never showed the standby. With it, each node answered 200 with its own values (vault_core_active 0 on a standby). Vault's /sys/metrics API page says follower nodes ignore metric requests, typically a timeout; on 2.2.1 the answer was a 307, not a timeout (page read 2026-11-06). Autopilot gauges come from the active node only, which reports every node's health (vault_autopilot_node_healthy, one series per node).
• Vault's default audit file mode is 0600 (a second file device enabled without mode was created -rw-------), which a Collector running as another user cannot read. The lab sets mode=0644; on a host, putting the agent in the vault group with 0640 is the narrower choice.
• Everything sensitive is HMAC'd, consistently. KV values, tokens, accessors, ciphertext, certificates and request data are hmac-sha256:…. The same token has the same HMAC everywhere, so issuance (vault.issued.accessor) joins to every later use (vault.auth.accessor) without unhashing anything.
• A PKI serial is HMAC'd too, unless the mount is tuned. vault secrets tune -audit-non-hmac-response-keys=serial_number pki/ makes the issued serial readable; serials are public, and this is what makes a certificate issuance history useful.
• The CLI adds preflight calls. Each vault kv get first reads sys/internal/ui/mounts/<path>: 24 preflight responses for 17 KV reads in this run. They are tagged preflight, not dropped.
• Enabling the device leaves two halves: Vault's own test request (sys/audit/test, no response) and the response to the enable call, whose request came before the device existed.
• Dashboard filters in SQL, not Lucene. A tile filtered with vault.event.category:denied (Lucene, a dotted attribute name) ended in "Error loading chart"; LogAttributes['vault.event.category'] = 'denied' drew.
• Login responses have no policy result (vault.policy.allowed is empty): the request had no token.
• Without file_storage, a Collector restart sends the whole file again: 274 lines became 548 rows (S7). With it, a restart neither lost nor duplicated an entry.
• Size: 274 rows took 42,195 bytes compressed in otel_logs (154 B per row; the original line is about 1.5 KB). A small sample — measure on real traffic before sizing.

Run it locally

Needs Docker (the stack used 1.8 GB of memory, measured 2026-11-06) and Python 3.

cd labs/vault-audit
bin/up.sh                  # 3 Vault nodes, init + unseal, ClickStack, Collector, HyperDX sources
bin/traffic.sh 3 vault-2   # three rounds, sent through a standby
bin/check.py               # S1–S8, about 3 minutes; stops and restarts the active node once
docker compose down -v

HyperDX: http://localhost:48080 (the test user is in .run/hyperdx-user). Screens from this stack, dark mode, 2026-11-06: images/en/, images/ko/. To take them again (needs the playwright package and its Chromium; cwebp for the WebPs):

bash PLAYWRIGHT=<path to the playwright package> node bin/capture.mjs all Vault nodes: 48201–48203. A restarted Vault node comes back sealed: run bin/vault_ha.sh again.

Managed ClickStack

The Collector writes to the Cloud service directly; Vault and the Collector stay where they are. bin/cloud.sh <env-file> does steps 1, 2 and 4 in one go from a file that sets CH_HOST, CH_USER and CH_PASSWORD, and refuses a database that already has tables.

1. cp .env.example .env, fill in the service host, user and password (chmod 600 .env).
2. Start the Collector on the new target with fresh read offsets, so it sends the existing files too: docker compose rm -sf otelcol && docker volume rm vaultlab_otelcol-storage && docker compose up -d --no-deps otelcol
3. In HyperDX on the Cloud service, add a Log source on vault_audit.otel_logs and a Metric source on vault_audit (otel_metrics_gauge, _sum, _histogram, _summary), with the default OTel column mappings, named "Vault Audit" and "Vault Metrics", then import dashboard/vault-audit.json (or .ko.json). The Log source's Timestamp Column must stay Timestamp: the form fills it in after the table is picked, and the contrib exporter's otel_logs has no TimestampTime column (nor does local ClickStack 2.42.0's own). With TimestampTime, every log tile answered Unknown expression or function identifier (2026-11-06). The metrics tiles start when the Collector was pointed at the service: the audit files are sent again from the start, a Prometheus scrape has no history.
4. set -a; . ./.env; set +a; bin/check.py — the same checks over HTTPS, without the S7 negative (its scratch Collector writes to the local ClickStack).

Tuning knobs

Knob	Where	Effect
CH_TTL	.env / compose	retention of rows (default one year here); set what the audit rule says
audit_non_hmac_response_keys, audit_non_hmac_request_keys	vault secrets tune per mount	fields written in clear; only for values that are not secret. Tried once on 2026-11-06: serial_number (response) is the lab's setting; common_name (request) put the CN in clear
hmac_accessor=false	vault audit enable file …	accessors in clear, for vault token lookup -accessor straight from a row. Tried once on a second device on 2026-11-06 (the accessor was in clear); not the lab's setting
category rules	transform/vault	extend for other engines (database, aws, ssh)
Prometheus keep list	prometheus/vault	which metric families are stored

Not covered

Vault Enterprise features (performance standbys, audit filtering, namespaces beyond root), TLS, auto-unseal, the socket and syslog audit devices, dynamic database credentials, and Vault running outside Docker.

 vault-1 ─┐ audit.log (노드별 볼륨)        ┌─ file_log ─ transform (OTTL) ─┬─ clickhouse ─▶ vault_audit.otel_logs
 vault-2 ─┼──────────────────────────────▶│                                └─ count ──┐
 vault-3 ─┘ /v1/sys/metrics (노드별)       └─ prometheus ───────────────────────────────┴▶ vault_audit.otel_metrics_*
                                         otelcol (여기선 하나, 운영에서는 Vault 호스트마다 하나)

cd labs/vault-audit
bin/up.sh                  # Vault 3노드, 초기화 + 봉인 해제, ClickStack, Collector, HyperDX 소스
bin/traffic.sh 3 vault-2   # standby를 거쳐 3회차
bin/check.py               # S1–S8, 약 3분. active 노드를 한 번 멈췄다 다시 띄웁니다
docker compose down -v