labs/privatelink-gateway

A validation plan for one topology: OpenTelemetry from many VPCs in several AWS accounts, sent to ClickHouse Cloud through one gateway collector and one AWS PrivateLink endpoint, with the service closed to the public internet. The point is to measure it before a team builds it in its own accounts, so the configuration they are handed has been run once.

Last run: 2026-10-14, terraform apply → bin/setup.sh → checks → terraform destroy in ap-northeast-2, one AWS account (Results). Changed since, without a re-run: K2's wait for gw-default's queue to drain before counting, and the consumers' earliest start in terraform/user_data.sh.tftpl on a freshly created cluster (the second K1–K4 run used earliest through bin/check.sh), #106. V8's overload control (#108) and K7 have not run.

The topology

service VPC · account A ─┐   apps: OTel SDK (OTLP) · agent collector scraping Prometheus exporters
service VPC · account B ─┼─→ Transit Gateway
other service VPCs ──────┘        │
                                  ▼
               central VPC ┌─────────────────────────────────────────────────────┐
                           │ internal NLB :4317/:4318                            │
                           │   → gateway collector (queue · retry · batch)       │
                           │   → interface VPC endpoint (the only way out)       │
                           └─────────────────────────────────────────────────────┘
                                  │  AWS PrivateLink
                                  ▼
               ClickHouse Cloud service — endpoint ID on its allow list, IP access list: deny all

The conditions it is built for: a private-only policy, Transit Gateway already joining the accounts, about ten VPCs in two accounts, and all three signals (metrics, logs, traces). Some teams already instrument with the OTel SDK or Prometheus exporters, and others are new to OpenTelemetry.

What the documentation already settles

Read on 2026-10-14. These are vendor documentation, not measurements; the plan below checks the ones the topology depends on.

Fact	Source
PrivateLink is available on the Scale and Enterprise plans	AWS PrivateLink
Steps: get endpointServiceId and privateDnsHostname (console or the privateEndpointConfig API), create an interface endpoint, add its endpoint ID to the service's allow list	same page
DNS is the customer's: point privateDnsHostname at the endpoint's network interfaces. The page's CloudFormation and Terraform examples turn the endpoint's own private DNS off	same page
Security group: ports 443, 8443, 9440, 3306	same page
Cross-region endpoints are supported for the listed regions, not for ap-east-2 or mx-central-1. AWS charges cross-region transfer	same page
PrivateLink carries only connections initiated from your VPC to ClickHouse Cloud	same page
IP access lists apply only to public-internet connections, not PrivateLink. To accept PrivateLink only, set the list to deny all. IPv4 only. Default: allow from anywhere	IP access lists
A collector of your own sends to Managed ClickStack. The ClickStack distribution is configured by CLICKHOUSE_ENDPOINT (the full HTTPS URL with port 8443), user and password. Use a dedicated ingestion user. Batch at least 1,000 rows (the example uses 10,000)	ClickStack: OTel collector
ClickHouse exporter: traces and logs beta, metrics alpha. Batch inside sending_queue.batch rather than the batch processor. The queue is in memory unless sending_queue.storage names a storage extension. retry_on_failure.max_elapsed_time defaults to 300 s	clickhouseexporter README on main. Check it against the collector version you actually run

The docs leave these open, and the plan measures them:

• Whether the Cloud console's SQL console and Managed ClickStack still query the service when its IP access list denies all. The IP access list page does not say. If they don't, people need a path to the UI that the topology above does not have. Measured (V9): both still query it.
• What happens to data when ClickHouse is unreachable for longer than the 300 s retry default. Measured (V7, K2): the file-storage gateway kept everything; the default gateway lost what it was sending when the 300 s ran out.
• How much load one gateway collector carries. Measured (V8): more than one t4g.small generator could send; the limit was not found.

Checks

Every check has a positive and a negative control. Data checks are SQL against otel_traces, otel_logs and otel_metrics_*, by a verify.run_id resource attribute the way _base/bin/verify.sh does it. A UI screenshot is not a result.

#	What	Positive	Negative
V1	Endpoint and allow list	from the central VPC, SELECT 1 over privateDnsHostname:8443 succeeds	remove the endpoint ID from the service's allow list: the same query fails (record the error)
V2	DNS stays in the central VPC	privateDnsHostname resolves to the endpoint's private IPs in the central VPC. The NLB's DNS name resolves to private IPs from a service VPC	from a service VPC, privateDnsHostname does not resolve to the endpoint's IPs, so apps never need that record
V3	Public path closed	IP access list set to deny all: V1 still succeeds	the public hostname from a host on the internet fails
V4	Through Transit Gateway	telemetrygen in a service VPC of the other account sends to the NLB, and the rows arrive	remove the security-group rule for 4317: no rows with that run_id arrive (telemetrygen still exits 0, so the count is the evidence)
V5	All three signals	N traces, N logs, N metric points with one run_id: each table counts N for that run_id	a run_id that was never sent counts 0 in each table
V6	Prometheus path	an agent collector scrapes an exporter in a service VPC, and its series appear in otel_metrics_gauge / otel_metrics_sum	stop the exporter: no new points with a value after the stop time (the first failed scrape writes one staleness marker per series, Flags bit 1)
V7	ClickHouse unreachable for longer than 300 s	gateway with a file-storage queue and a raised max_elapsed_time: send N during the outage, restart the gateway once mid-outage, restore. The count reaches N	the same with defaults (memory queue, 300 s): the count stays below N. Record how far below
V8	Load and delay	at a fixed rate for 10 minutes: the time from the generator's last send to the count reaching N, and the gateway's CPU and memory from its own telemetry (:8888)	a run at a rate the gateway cannot sustain: the queue grows (record where, and what is dropped)
V9	People can reach the UI	with the IP access list at deny all, run a query in the SQL console and open the Managed ClickStack search. Record what happens	the same with an allow entry for the tester's IP, to tell an access-list effect from anything else

V7 is the outage the topology must survive. The central VPC's one endpoint is the only way out, so anything that breaks it stops every team at once.

Option: a Kafka buffer

The gateway's file-storage queue covers a short outage and a restart (V7). It lives on one node, so it does not cover the loss of that node, an outage longer than its disk holds, replaying data, or a second consumer. If any of those becomes a requirement, put Kafka (Amazon MSK) in the central VPC.

service VPCs ─→ Transit Gateway ─→ central VPC
                                   internal NLB :4317/:4318
                                     → ingest collector    OTLP in → kafka exporter (otlp_proto)
                                     → MSK                 otlp_spans · otlp_logs · otlp_metrics
                                     → consumer collectors kafka receiver → clickhouse exporter
                                     → interface VPC endpoint → PrivateLink → ClickHouse Cloud

Apps keep the same OTLP endpoint, and the only connection to ClickHouse Cloud is still the one the VPC opens. What the components' READMEs say (contrib main, read 2026-10-14; check against the version you run):

• The Kafka exporter and receiver are beta for traces, metrics and logs. Encoding defaults to otlp_proto, and topics default to otlp_spans, otlp_logs and otlp_metrics.
• The receiver's message_marking.after defaults to false: a message is marked consumed before the pipeline runs, so a consumer that dies mid-export loses it. Set it to true for at-least-once, which means duplicates after a restart.
• initial_offset defaults to latest, and group_id to otel-collector.

Not the first choice: ClickPipes pulling from MSK. ClickPipes can read MSK privately through a reverse private endpoint over MSK multi-VPC connectivity, which is not cross-region (ClickPipes PrivateLink, read 2026-10-14). That removes the consumer collectors, at two costs:

• It adds a second private connection, opened from the ClickHouse side into the MSK cluster, which the private-only policy has to approve separately.
• The ClickPipes Kafka reference lists JSON, AvroConfluent and Protobuf, does not mention OTLP, and nests Tuple and Array one level deep only. OTLP JSON nests deeper (resource → scope → record), so it would land raw and need materialized views into the otel_* schema. That is unmeasured and kept as a hypothesis (K7).

Kafka checks (run after V1–V9, on the same environment plus MSK):

#	What	Positive	Negative
K1	Same data through Kafka	V5 through the Kafka path: N per signal for one run_id	a run_id never sent counts 0
K2	Outage longer than the gateway's disk would hold	ClickHouse unreachable for an hour: consumer lag grows, then drains on restore, and the count reaches N	V7's file-storage gateway, same outage: record what it kept and what it lost
K3	Consumer dies mid-export	message_marking.after: true, kill the consumer while it exports: the count reaches at least N. Record the duplicates (count − uniqExact of the record key)	message_marking.after: false, same kill: the count stays below N
K4	Replay	reset the consumer group to earliest into a separate database: its counts equal the original's for the run_id	a consumer group with initial_offset: latest started after the send reads nothing of it
K5	Losing one consumer	two consumers in the group, stop one: rebalance, the count reaches N	— (K3 covers loss)
K6	Spike	a burst above the insert rate: lag grows, then drains. Record peak lag and the time to drain	the same burst without Kafka (V8's gateway): record where it queues or drops
K7	ClickPipes variant (hypothesis)	otlp_json topics through a ClickPipe over the reverse private endpoint into a raw table, then materialized views into otel_*: counts equal K1's	— run only if K1–K6 make Kafka a requirement

Kafka is worth it when K2, K3 or K4 meets a requirement the gateway's file-storage queue cannot (V7). Otherwise it is one more cluster to run.

Test environment

The smallest build that answers the checks. Everything is in one region, the same as the Cloud service. A cross-region endpoint is a separate question.

Part	What	Why
ClickHouse Cloud	one Scale service, a database and an ingestion user of its own	PrivateLink needs Scale or above
Central VPC	interface endpoint in two AZs, a Route 53 private hosted zone for privateDnsHostname associated with this VPC only, an internal NLB, a gateway collector on one EC2 instance	V1–V3, V7, V8
Service VPC, same account	telemetrygen, an agent collector, one Prometheus exporter	V5, V6
Service VPC, second account	telemetrygen only	V4: the Transit Gateway shared across accounts. One account proves the routing but not the sharing
Transit Gateway	attachments for the three VPCs	V4
MSK, Kafka checks only	the smallest provisioned cluster in the central VPC, an ingest collector, two consumer collectors	K1–K6. K7 adds a ClickPipe and its reverse private endpoint

Collector images and telemetrygen are pinned when the lab is built and recorded in every result line, with the ClickHouse version the service reports.

Cost before running. Line items to estimate from the AWS Price List and the ClickHouse Cloud pricing page, for the hours the run needs: - the Scale service - the interface endpoint (per AZ-hour and per GB) - the NLB - the Transit Gateway (per attachment-hour and per GB processed) - the EC2 instances - cross-AZ transfer - for the Kafka checks: MSK broker-hours and storage. For K7: ClickPipes compute

Estimated before the 2026-10-14 run (AWS Price List API for ap-northeast-2, read that day):

Item	Rate	Per hour
ClickHouse, 2 compute units (2 × 8 GiB)	the pricing page lists no Scale rate for Seoul; its Enterprise rate, $0.4801 per unit-hour, as a ceiling	≤ $0.960
Transit Gateway, 3 VPC attachments	$0.07 each	$0.210
EC2 t4g.small × 3	$0.0208 each	$0.062
NLB	$0.0225 + LCU	≈ $0.030
Interface endpoint, 2 AZs	$0.013 each	$0.026
Public IPv4 × 3	$0.005 each	$0.015
Base		≈ $1.31
Kafka: MSK kafka.t3.small × 2, 2 more instances and addresses, storage	$0.0569 per broker	≈ $0.17 more

Data processing ($0.01/GB at the endpoint, $0.02/GB at the Transit Gateway) came to cents at these volumes. The actual run is under Results. Nothing is launched without the owner's go-ahead.

How to run

Ran end to end on 2026-10-14 (Results). Before that, terraform validate and otelcol-contrib 0.155.0 validate passed and each failed when a fault was injected, and the SQL and the V5 and V7 controls ran once against a local ClickHouse 26.9.7.9 container.

Path	What
terraform/	the ClickHouse service and ip_access, three VPCs and the Transit Gateway, the endpoint and its private hosted zone, the NLB, EC2 with SSM, and MSK behind enable_kafka
collector/	gateway-durable (file-storage queue, long retry), gateway-default (defaults), agent, ingest-kafka, consumer-kafka — otel/opentelemetry-collector-contrib:0.155.0
bin/	setup.sh (ingest user and databases), check.sh <V1…V8, K1…K6>, outage.sh block/unblock
sql/	00-ingest-user.sql (the ingest user's grants, sent by setup.sh), counts.sql and dupes.sql (row counts and duplicate spans per run, used by check.sh)
results/	each check.sh run writes its result lines to <UTC date>-<id>.txt here

export AWS_PROFILE=<profile>
export CLICKHOUSE_CLOUD_API_KEY=<key>
export CLICKHOUSE_CLOUD_API_SECRET=<secret>
export CLICKHOUSE_ORG_ID=<org id>
cd labs/privatelink-gateway/terraform
cp terraform.tfvars.example terraform.tfvars    # set clickhouse_org_id
terraform init && terraform plan -out=tf.plan && terraform apply tf.plan
cd ../bin && ./setup.sh && ./check.sh V1        # … V8, then enable_kafka = true and K1 … K6
cd ../terraform && terraform destroy

All SQL runs on the gateway through SSM, because the service accepts nothing from the public internet. There is no SSH and no NAT. Instances take a public IP only to pull packages and images.

Found while building, and kept in the plan: - create_schema: true stops a collector from starting while ClickHouse is unreachable. In V7 a gateway restarted mid-outage cannot accept OTLP until the outage ends; its persisted queue still replays afterwards. A production gateway should create the schema once and run with create_schema: false. - A generator's own count is not evidence. telemetrygen v0.155.0 emits two spans per trace even with --child-spans 0, counts --rate per worker, and at an unthrottled rate reported far more traces than arrived at a gateway that refused nothing. The checks count what the receiver accepted. - One AWS account. V4 proves Transit Gateway routing between VPCs, not sharing it across accounts. - Passwords reach the instances through user data and SSM commands. That is acceptable for a lab torn down the same day, not for anything kept.

Found while running (2026-10-14), and fixed where it says so: - ClickHouse Cloud rejects a SQL user's password without a special character (CREATE USER, Code 36, "contain at least 1 special character"), while the service API took the alphanumeric default password. bin/setup.sh stopped there. Fixed: random_password.ingest now has two specials from -_.~. - ip_access = [] is stored as [] and means deny all: the public hostname reset the connection (curl: (35)) and, with the tester's /32 added, the same request from the same machine returned 200 (V3, V9). - A revoked and re-authorized security-group rule comes back with a new ID, so the next terraform plan wants to create it again, and the apply would fail on a duplicate. Before each later plan the rule was removed from state and imported by its new ID. terraform destroy is not affected (#107). - create_schema: true measured: the gateway restarted mid-outage failed to start 10 times (create database: failed to query server hello) and started 15 s after the outage ended (V7). - A full memory queue refuses at the receiver: gw-default accepted exactly its 1000 requests and refused the rest, so accepted never exceeds the queue. Requests waiting in the queue did not age: after a 600 s outage it delivered 4910 of 5000 accepted traces. With 200 traces, all of them in flight at once, it delivered none (V7). That the 300 s clock starts only when a consumer takes a request is read from these numbers, not from the exporterhelper source. K2's control was rewritten to use the refused count. - telemetrygen retries refused batches past --duration; a synchronous send to the full gw-default held past the SSM limit. K2 now runs the gateway sends detached and removes them after the send window. - Kafka consumers with initial_offset: latest skip data on a partition with no committed offset. The topics are auto-created by the first produce, after the consumers join: K1 counted 0 of 200 per signal, K3's positive lost half, and K4's original group counted 0 logs and 0 metrics while the replay group (earliest) read 200 of each. Fixed: the default group starts from earliest, which applies only where it has no committed offset. - The Prometheus receiver writes a staleness marker per series at the first failed scrape (363 points, Flags bit 1, no value). V6's negative now counts only points with a value. - One t4g.small telemetrygen tops out near 8,700 spans/s (asked for 20,000). V8's overload control needs more generators than this build has.

Results

Run on 2026-10-14 in ap-northeast-2, one AWS account. Versions, as each result line records them: ClickHouse Cloud 26.6.1.2326 (SELECT version()), a service of 2 replicas × 8 GiB with idle scaling; collector otel/opentelemetry-collector-contrib:0.155.0; telemetrygen v0.155.0; prom/node-exporter:v1.12.1; MSK Kafka 3.9.x on 2 × kafka.t3.small; Terraform 1.16.4 with hashicorp/aws 6.68.0 and ClickHouse/clickhouse 3.38.0. Managed ClickStack was used through the console; its version is not shown there and was not recorded.

Verified on ClickHouse Cloud 26.6.1.2326, Managed ClickStack (version not recorded) and otel/opentelemetry-collector-contrib:0.155.0, 2026-10-14: V1–V7, V8 positive, V9, K1–K6.

#	Result	What was measured
V1	PASS	SELECT 1 over the private hostname from the central VPC. Endpoint ID removed from the allow list (API 200): curl: (35) Recv failure: Connection reset by peer. Re-added: works again
V2	PASS	central VPC: the private hostname resolves to the endpoint's two IPs. svc_a: the NLB name resolves to 10.60.x addresses; the private hostname resolves to nothing
V3	PASS	API reports ipAccessList: []; PrivateLink still answers. Public hostname from the tester's machine: curl: (35) connection reset, and HTTP 200 once that machine's /32 was allowed (V9)
V4	PASS	200 of 200 traces from svc_b through the Transit Gateway and the NLB. 4317 rule revoked: 0 rows (telemetrygen exit 0). Rule back: sends again
V5	PASS	one run_id: 200 traces (400 spans), 200 logs, 200 gauge points. A never-sent run_id: 0 in each
V6	PASS, second run	240 gauge and 107 sum metric names from node-exporter. After the stop: 0 points with a value, 363 staleness markers. The first run counted the markers and failed
V7	PASS	outage 360 s. gw-durable, restarted once mid-outage: 200 of 200 (400 spans, 0 duplicates). gw-default: 0 of 200
V8	positive PASS, negative not reached	2000 spans/s for 600 s: 599,335 traces accepted and counted, the full count 16 s after the last send; gateway (t4g.small) 2–7 % CPU, 68–74 MiB container memory, peak queue 3,700 items. Asked for 20,000 spans/s: 523,139 traces in 120 s (≈ 8,700 spans/s), gateway ≈ 18 % CPU, peak queue 11,500 items, nothing refused or lost
V9	PASS	IP access list []: the console's SQL console ran SELECT version() → 26.6.1.2326, and Managed ClickStack searched a Logs source on otel_pl.otel_logs: 800 results, the same 800 rows SQL counts. With the tester's /32 allowed, the public hostname answered from that machine
K1	PASS, second run	200 of 200 per signal through Kafka; a never-sent run_id 0. First run 0 of 200 per signal (latest, above)
K2	PASS	outage 600 s, not the planned hour. Kafka path: 12,625 of 12,625. gw-default: 5000 accepted, 70 spans refused, 4910 delivered once its queue drained (the check counted 3315, 30 s after the restore). gw-durable: 19,516 of 19,516
K3	PASS, second run	after: true, both consumers killed during an outage: 4000 of 4000 spans, 0 duplicates. after: false: 1000 of 4000, 3000 lost
K4	PASS, second run	replay group from earliest into otel_replay: 200 traces, logs and gauge points, equal to the original. A latest group started after the send: 0, then read its marker
K5	PASS	consumer-1 stopped while holding unexported messages: consumer-2 delivered 2000 of 2000 after the rebalance
K6	PASS	60 s burst through Kafka: 258,483 traces, peak backlog 129,471, the full count 92 s after the last send. The same burst at the gateway: 260,519 traces, peak queue 11,500 items, nothing refused or lost
K7	not run	hypothesis; run only if K1–K6 make Kafka a requirement

What this says about the topology. One PrivateLink endpoint with the IP access list at deny all carries all three signals from both service VPCs, and people keep the console and Managed ClickStack. The gateway's file-storage queue survives an outage longer than 300 s and a restart, so it, not the defaults, belongs in the handover. At the load one generator could make, the gateway was not the bottleneck, and Kafka added delay (92 s to drain a burst the gateway absorbed) without a requirement that the file-storage queue failed.

Cost: the base build ran 06:34–09:37Z (3.05 h) and the Kafka part 07:17–09:37Z (2.33 h). At the rates under Test environment that is 3.05 × $1.31 + 2.33 × $0.17 ≈ $4.39, plus cents of data processing. The ClickHouse rate is the Enterprise ceiling and the service scaled idle when unused, so the bill should be lower. Destroy removed 95 resources; a search by name and tag afterwards found none left.

---

서비스 VPC · 계정 A ─┐   앱: OTel SDK(OTLP) · Prometheus exporter를 scrape하는 agent Collector
서비스 VPC · 계정 B ─┼─→ Transit Gateway
나머지 서비스 VPC ───┘        │
                              ▼
               중앙 VPC ┌─────────────────────────────────────────────────────┐
                        │ 내부 NLB :4317/:4318                                │
                        │   → gateway Collector (queue · retry · batch)       │
                        │   → interface VPC endpoint (유일한 출구)             │
                        └─────────────────────────────────────────────────────┘
                              │  AWS PrivateLink
                              ▼
               ClickHouse Cloud 서비스 — 허용 목록에 endpoint ID, IP access list는 deny all

서비스 VPC ─→ Transit Gateway ─→ 중앙 VPC
                                 내부 NLB :4317/:4318
                                   → ingest Collector     OTLP 수신 → kafka exporter (otlp_proto)
                                   → MSK                  otlp_spans · otlp_logs · otlp_metrics
                                   → consumer Collector   kafka receiver → clickhouse exporter
                                   → interface VPC endpoint → PrivateLink → ClickHouse Cloud

export AWS_PROFILE=<profile>
export CLICKHOUSE_CLOUD_API_KEY=<key>
export CLICKHOUSE_CLOUD_API_SECRET=<secret>
export CLICKHOUSE_ORG_ID=<org id>
cd labs/privatelink-gateway/terraform
cp terraform.tfvars.example terraform.tfvars    # clickhouse_org_id 입력
terraform init && terraform plan -out=tf.plan && terraform apply tf.plan
cd ../bin && ./setup.sh && ./check.sh V1        # … V8, 그다음 enable_kafka = true로 K1 … K6
cd ../terraform && terraform destroy