Migrating from Elastic

An Elastic migration splits into three parts, and which one dominates varies enormously between deployments. A team that ran its own pipeline into Elasticsearch has no ingest work at all; a team on Elastic Agent has a great deal. So these are three independent parts, each usable on its own, and the first useful thing this lab does is let you establish which ones you can skip.

Part	Elastic input	ClickStack output	Status
ingest	GET _ingest/pipeline, Beats or Logstash config, index templates	a collector config fragment	planned, ingest/
data	GET _mapping plus the index contents	ClickHouse DDL and a load	written, data/
identity	ids that differ between the two systems	a mapping table, a dictionary and a quarantine	written, data/idmap/
dashboards, alerts	Kibana saved objects, alerting rules	HyperDX dashboards and alerts	separate, under skills/

Two of the three already have a destination in this repository: otel-profiles for collector configuration and clickstack-config for sources, dashboards and alerts. So the new surface is smaller than "three parts" suggests — the data path, plus converters that feed assets that already exist.

Driving this with an agent

Most people running this will be driving it with a coding agent, the same way it was built. AGENTS.md in this directory is written for that reader rather than for a contributor: where to look for each question, the order the steps go in, which two steps are human decisions, the invariants whose failure mode is a migration that looks finished, and the questions an agent should put to a person instead of answering by default.

It opens with what this lab does not do -- schema design, the scale it was actually verified at, the Cloud-specific path, an index pattern whose mappings disagree -- because the lab's own rule is to refuse plausible output for a case that did not convert, and that has to apply to the documentation too.

Read it before the sections below if you are running a real migration. Read the repository root's AGENTS.md instead if you are changing this lab.

Start with the official documentation

ClickHouse documents this migration already, and this lab does not restate it:

• Migrating to ClickStack from Elastic
• Equivalent concepts
• Mapping types
• Migrating data
• Migrating agents

Read those first. What is left here is what they do not cover, and the largest gap is one the data page states about itself: export over JSON and HTTP is only viable below roughly ten million rows, because that is what the scroll API and elasticdump comfortably sustain. Real observability datasets start above that ceiling, so the documented path does not apply to the cases that most need help.

All three parts have the same shape

Parse an Elastic artifact, emit a ClickStack artifact, then compare the two sides. The third step is the one that matters, and it is the same idea every time — a pair of queries, one per system, that must agree.

Part	Conversion	Check
ingest	ingest processors to OTTL or filelog operators	push one identical input line through both, compare the resulting fields
data	_mapping to DDL	document counts per time bucket, and field-level sampling
dashboards	panel query to SQL	the same time window must produce the same number

A conversion without its pair is a claim, not a result. This is also what lets a tool grade its own output rather than asking someone to eyeball a dashboard.

Conversion is not one to one, and saying so is the feature

No part of this maps cleanly, and the failure mode to design against is emitting something plausible for the cases that did not convert. Every converter here classifies each item it handles:

• converted — a direct equivalent exists
• needs review — an equivalent exists but behaviour differs at the edges, such as grok, which has no direct OTTL counterpart and has to be expressed as a regex or an operator
• unsupported — no mechanical conversion, such as a script processor

A silent guess in the second or third category is worse than a refusal, because it is discovered in production rather than during the migration.

Questions that change the size of the work

Worth settling before starting. The answers can shrink this considerably:

• total volume: index count, document count, size on disk
• whether everything moves, or only the retention window, leaving the old cluster read-only for the rest
• whether both systems run in parallel for a period, which adds dual-write

Order

data/ first — it is the only part with no official coverage. The other two attach to assets that are better left to settle, and are simply absent for some migrations.

Tracked as issues so a part a given migration does not need can be skipped rather than worked around.

---