Vector → ClickStack next to an existing Loki pipeline

Verified on Vector 0.61.0 (Helm chart 0.61.0), clickstack-all-in-one 2.44.0 (ClickHouse 26.8.36.3), Loki 3.7.9, Kong 3.9.4, on k3d (k3s v1.35.5), 2026-11-13.

The question. Vector already ships every Kubernetes log to Loki. Can the same Vector also send some workloads to ClickHouse, into their own tables, while the Loki path stays as it is? What happens to Loki while ClickHouse is down? Here two Kong gateways (kong-external, kong-internal) and one API deployment are the workloads to route. A traffic generator stands in for everything else.

Layout

        kubernetes_logs (k8s)  ── every pod on the node
         │                    │
         │ (existing)         │ (added)
         ▼                    ▼
   loki_labels ─▶ loki   by_workload (route)
                          ├─ kong_external ─┐
                          ├─ kong_internal ─┴▶ kong_rows (remap) ─▶ ch_kong ─▶ logs.kong_access
                          ├─ api ─────────────▶ api_rows  (remap) ─▶ ch_app  ─▶ logs.app_logs
                          └─ _unmatched        (no consumer: Loki only)

• vector/values.yaml: the whole Vector config. The Loki half stays as it was. The route, the two remaps and the two ClickHouse sinks are the additions.
• vector/mode-*.yaml: the ClickHouse sinks' buffer. block is the default when_full; drop uses drop_newest; disk uses a 256 MiB disk buffer with drop_newest. The Loki sink keeps Vector's default buffer in every mode.
• sql/schema.sql: logs.kong_access (gateway fields as columns, Gateway = external/internal, Cluster) and logs.app_logs (JSON unpacked, a non-JSON line kept whole in Body).
• k8s/: Loki, ClickStack (one PVC for ClickHouse and the UI's MongoDB), two Kong Deployments with the same DB-less config told apart by the pod label app, and the API.
• bin/lab.py: up, mode, run <RUN> <N> (N requests to each gateway, tagged run=<RUN>), counts, outage, down.

Run

bin/lab.py up                       # k3d cluster, backends, workloads, schema, Vector (block)
bin/lab.py run R3 300 && sleep 20 && bin/lab.py counts R3
bin/lab.py outage block B3 3000     # ClickStack scaled to 0, 3,000 requests, 2 min, back up
bin/lab.py outage drop  D2 3000
bin/lab.py outage disk  K1 3000
bin/lab.py down

The UI is at kubectl --context k3d-vector-lab -n clickstack port-forward svc/clickstack 8080:8080. Use port 8080: after login the UI redirects to localhost:8080.

Results (2026-11-13)

Routing and dual write (results/routing-R3.txt): 300 requests to each gateway.

	ClickHouse	Loki
kong-external	300 (Gateway = external)	300
kong-internal	300 (Gateway = internal)	300
API (one line per request)	600 (12 ERROR, 588 INFO)	600
traffic generator (unmatched)	0 in either table	300

Each gateway had 6 responses with status 500, as designed (every 50th request).

ClickHouse down for about 2 minutes: 3,000 requests to each gateway, so 6,000 rows per ClickHouse sink. Logs: results/outage-*.txt.

ClickHouse sink buffer	Loki during the outage	ClickHouse after recovery	Discarded
memory 500, block (default)	stopped at 1,959 of 3,000 for 124 s. The unrelated generator's logs stopped too	6,000 / 6,000 kong, 6,000 / 6,000 API, 31 s after restart	0
memory 500, drop_newest	complete (3,000 / 3,000) within 20 s	2,816 / 6,000 kong, 2,500 / 6,000 API	3,184 + 3,500, exactly the shortfall
disk 256 MiB, drop_newest	complete within 20 s	6,000 / 6,000 kong, 6,000 / 6,000 API, 31 s after restart	0

What this shows:

1. block stops the whole source, not just the ClickHouse branch. Loki got nothing new for as long as ClickHouse was down, including logs from pods that never go to ClickHouse. Nothing was lost in this run, because kubernetes_logs resumes from its file checkpoints. The cost is a Loki gap for the length of the outage. Log rotation during a long outage was not tested.
2. drop_newest keeps Loki whole and sacrifices ClickHouse. vector_buffer_discarded_events_total counted exactly what ClickHouse was missing.
3. A disk buffer gives both for as long as it lasts. The 6,000 gateway rows took 13.4 MB in the buffer (about 2.2 KB per row, raw line included), so 256 MiB holds about 120,000 such rows. That figure is computed, not measured: hold time = buffer size ÷ the routed rate.

One bad row drops the whole batch. On the first run, Kong 3.9.4 logged latencies.proxy = -3 for one request. The table had UInt32, so ClickHouse rejected the 1,000-row insert (CANNOT_PARSE_NUMBER, HTTP 400). Vector treats a 400 as not retriable and dropped all 1,000 rows (vector_component_discarded_events_total{intentional="false"}, results/outage-drop-D1-uint-schema.txt). The latency columns are now Int32. Watch that counter, not only the buffer counters.

Other things seen on the way:

• A new pod's log file can take up to glob_minimum_cooldown_ms (default 60,000 ms) to be picked up; the lab sets 5,000.
• Helm passes customConfig through tpl, so Vector's {{ field }} templates must be written {{ "{{" }} field {{ "}}" }}.
• async_insert is on by default in this ClickHouse build (system.settings: async_insert = 1, changed = 0). Vector's 1,000-row batches went through it.

ClickStack. Both tables are ordinary log sources. app_logs (OTel-like column names) was inferred automatically. For kong_access, the timestamp, service (Service), body and log level (if(Status >= 500, 'error', 'info')) were set by hand.

Not covered

GCP Private Service Connect, ClickHouse Cloud regions and cross-region latency, a dedicated insert-only user (the lab uses the image's api user), and log rotation during long outages.

        kubernetes_logs (k8s)  ── 노드의 모든 파드
         │                    │
         │ (기존)             │ (추가)
         ▼                    ▼
   loki_labels ─▶ loki   by_workload (route)
                          ├─ kong_external ─┐
                          ├─ kong_internal ─┴▶ kong_rows (remap) ─▶ ch_kong ─▶ logs.kong_access
                          ├─ api ─────────────▶ api_rows  (remap) ─▶ ch_app  ─▶ logs.app_logs
                          └─ _unmatched        (소비자 없음: Loki에만 감)

bin/lab.py up                       # k3d 클러스터, 백엔드, 워크로드, 스키마, Vector(block)
bin/lab.py run R3 300 && sleep 20 && bin/lab.py counts R3
bin/lab.py outage block B3 3000     # ClickStack을 0으로 줄이고 요청 3,000개, 2분 뒤 다시 올림
bin/lab.py outage drop  D2 3000
bin/lab.py outage disk  K1 3000
bin/lab.py down