Self-hosted LLM serving observed with ClickStack

Related notes (Korean): 자가 호스팅 LLM 서빙 관측: OTel Collector에서 ClickStack까지

The OTel Collector configuration, the nvidia-smi → OTLP pusher and the HyperDX dashboard (v3, 32 tiles) that observed a vLLM coding model behind a LiteLLM gateway on one NVIDIA GB10 box, made runnable on a laptop with no GPU. vLLM and nvidia-smi are replaced by small stand-ins; the Collector configuration, the pusher and the dashboard are the ones from the article. Six scenarios are checked with SQL and the HyperDX API, each with a positive and a negative check.

The numbers in the article (GPU fields, data volume, decode impact) were measured on the GB10 box on 2026-09-10/11 and are not reproduced here: a mock has no GPU and no decode speed. This lab checks the configuration and the behaviour that does not depend on the hardware.

What it builds

Piece	Where	What
ClickStack all-in-one 2.42.0	docker-compose.yml	ClickHouse and HyperDX. The Collector writes to its ClickHouse as the image's published dev user worker (the image refuses default over the network)
OTel Collector contrib 0.160.0	otelcol/config.yaml	the article's configuration: prometheus scrape of vLLM with a keep filter, host_metrics with the utilization gauges named, snap's loop/squashfs excluded, otlp on all three pipelines, the ClickHouse exporter with create_schema into database llm_obs. Only the exporter's endpoint, user and password come from the environment
Mock vLLM	vllm-mock/mock.py	Prometheus /metrics with vLLM's metric names (two families the filter must drop) and an OpenAI-compatible /v1/chat/completions that moves the counters
LiteLLM v1.108.0	litellm/config.yaml	the gateway, OTel callback with traces, events and metrics, and the attribute allowlist. config-default.yaml is the same without the allowlist (S4 negative). A random master key is generated into .run/ by bin/up.sh
GPU pusher	gpu-push/gpu_push.py	the article's pusher, standard library only. Here it reads gpu-push/fake-nvidia-smi; on a GPU box drop the PATH line and add gpus: all
Dashboard	dashboard/llm-serving-v3.json	the article's dashboard: 8 containers, 32 tiles, sources named LLM Trace, LLM Metrics, LLM Log
Setup and checks	bin/up.sh, bin/setup_hyperdx.py, bin/check.py	stack up, a localhost HyperDX test user, the three sources, the dashboard import, and the scenario checks

Findings that shaped the lab

• The image's default sources cannot be reused. In 2.42.0 its Logs and Traces sources read the default database and ClickStack's rollup tables (otel_logs_kv_rollup_15m, otel_traces_kv_rollup_15m). The contrib exporter fills llm_obs and creates no rollup tables. setup_hyperdx.py copies the default sources' shape, points it at llm_obs and drops the rollup setting. Both databases' otel_logs have a single Timestamp column in this image.
• Dashboard import matches sources by name, case-insensitively. HyperDX's import page (DBDashboardImportPage.tsx, 2.42.0, same logic in 2.38.0) maps each tile's source name to a source of the same name and requires a manual mapping for any that does not match. setup_hyperdx.py does the same mapping through the API and stops on an unmatched name.
• LiteLLM 1.108.0 refuses to start without a master key ("no master key is set, so every request would be accepted without authentication"). bin/up.sh generates one into .run/.
• The OTLP HTTP body limit defaults to 20 MiB in Collector 0.160.0 (confighttp), not 4 MiB. The configuration keeps the article's 32 MiB.

Run it

cd labs/llm-serving-observability
bin/up.sh            # under a minute once the images are pulled
python3 bin/check.py # about 4 minutes
docker compose down -v && rm -rf .run

HyperDX is at http://localhost:38080; the test user is in .run/hyperdx-user. Ports: 38080 (UI), 38000 (HyperDX API), 38123 (ClickHouse HTTP), 34000 (LiteLLM), 38888 (Collector metrics), all on 127.0.0.1; S2 briefly uses 38318.

The checks

	Check	Negative check
S1 config	otelcol/config.yaml passes validate on Collector 0.160.0	exclude_devices under disk fails with 'diskscraper.Config' has invalid keys: exclude_devices
S2 GPU push	the five gpu.* gauges of service gb10-gpu are in llm_obs.otel_metrics_gauge	a Collector whose metrics pipeline lacks otlp answers /v1/metrics with 404 (200 with it)
S3 vLLM	vllm:kv_cache_usage_perc, vllm:prefix_cache_hits_total, vllm:time_to_first_token_seconds arrive with the colon; the utilization gauges named in host_metrics arrive	vllm:estimated_read_bytes_per_gpu, vllm:cache_config_info and an underscore spelling never arrive; samples scraped > samples after relabel
S4 gateway	with the allowlist, 10 requests leave 2 series of gen_ai.client.token.usage (input, output)	without it, the same 10 requests leave at least 10 (one attribute set per request and token type)
S5 traces	a marker sent through LiteLLM is in otel_traces (gen_ai.input.messages) and otel_logs	a marker sent straight to vLLM is in neither, while vllm:request_success_total still moves
S6 HyperDX	the dashboard's 32 tiles each map to one of the three llm_obs sources; the prefix-cache ratio computes from the same rows	the image's default Logs source reads default, which has no LiteLLM spans, and its rollup table does not exist in llm_obs

Verified on ClickStack all-in-one 2.42.0 (HyperDX 2.42.0, as reported by its /health; ClickHouse 26.8.34.13), OTel Collector contrib 0.160.0, LiteLLM 1.108.0, Python 3.12.16 — local Docker Desktop (macOS arm64), 2026-11-05: from an empty volume, bin/up.sh then bin/check.py, 14 of 14 checks passed, the negative checks included (S1 fault: exclude_devices; S4: 2 series with the allowlist, 20 without, after 10 requests each).

On a real box

Point CH_ENDPOINT / CH_USER / CH_PASSWORD at your ClickHouse (for ClickHouse Cloud, tcp://<service>.clickhouse.cloud:9440?secure=true and default), scrape your vLLM instead of the mock (static_configs.targets), run the pusher with --gpus all and without the fake nvidia-smi, and pin LiteLLM to a version you have checked. The GB10 notes behind this lab used tags (main-stable, latest), which is why their versions are unknown.

cd labs/llm-serving-observability
bin/up.sh            # 이미지를 받아 둔 뒤라면 1분이 안 걸림
python3 bin/check.py # 4분쯤
docker compose down -v && rm -rf .run