ClickHouse HOLs
GitHub

Resharding a self-managed ClickHouse cluster: 2 → 3 shards


ClickHouse OSS does not rebalance shards. The "Rebalancing Data" guide says so in its first sentence and lists four manual ways, in order of preference (read 2026-11-07 at ClickHouse/ClickHouse e73957f, docs/guides/oss/deployment-and-scaling/scaling-clusters.mdx). This lab runs them on a local cluster — ClickHouse Keeper, two shards of two replicas, then a third shard — and measures each one: rows per shard before and after, how long it took, what a background reader and writer saw during the move, and a per-partition check of what was copied. One failure case kills a source replica in the middle of a copy.

Verified on ClickHouse 26.8.36.3 (LTS), 2026-11-07 — one end-to-end run.py all started 13:44 KST, server and Keeper from the clickhouse/clickhouse-server:26.8.36.3 image, Docker Desktop VM with 12 CPUs and 7.7 GiB, no other containers running. Nothing here has been run on another version. Earlier runs the same day are in results/earlier-runs/.

Option What moves Run here
A. New shard + <weight> nothing; new writes are biased to the new shard yes
B. INSERT … SELECT from remote() into a new 3-shard layout every row, re-hashed yes (remote, not remoteSecure)
C. clickhouse-copier every row no — moved out of the bundle in 24.3, "no longer supported" (see below)
D. FETCH PARTITION + ATTACH + DROP (and MOVE PARTITION TO TABLE as a staging step) whole partitions yes

Run it

Needs Docker (about 3–4 GB free for the seven containers) and Python 3; no Python packages.

cd workload/resharding
python3 run.py up          # Keeper + 6 servers; HTTP on 127.0.0.1:18201-18206
python3 run.py all         # selftest, A, D, failure, B -> results/<version>.json and .log (about 17 min, 6 of them waiting on timeouts)
python3 report.py          # RESULTS.md and images/d-reads-<version>.svg from the JSON
python3 run.py down        # removes containers and volumes

python3 run.py a (or b, d, fail, selftest) runs one option from a fresh 2-shard state and writes results/<version>-<option>.json; python3 run.py resume b d runs steps into the existing results/<version>.json. ROWS=600000 python3 run.py all is a quick pass. CH_IMAGE=… picks another image for up.

Set-up

Results

Full tables: RESULTS.md. Rows per shard (shard 1 / 2 / 3, local count() on the first replica, including the prober's rows):

Option Before After Took Check
A. weights 1:1:2 + 3,000,000 new rows 6,002,487 / 5,997,513 / 0 6,755,667 / 6,745,787 / 1,500,946 config picked up in 0.92–1.14 s PASS
B. remote() copy, 6 partitions online + hot partition in a write freeze 6,002,487 / 5,997,513 / 0 4,017,368 / 3,973,248 / 4,014,084 2.6 s bulk, 0.31 s write freeze PASS
D. 2 partitions from each old shard to shard 3 6,002,487 / 5,997,513 / 0 4,077,796 / 4,041,946 / 3,884,658 0.10–0.30 s per partition PASS

reads during D

Failure: a source replica killed during a copy

docker kill rs-ch-s2r2 when the copy had read half of a partition. The copy read replica 2 of each shard first (remote('ch-s1r2|ch-s1r1,ch-s2r2|ch-s2r1', …), load_balancing = 'in_order') and wrote to replica 1, so the killed node was only a source. Copies were throttled to 2 MB/s (max_network_bandwidth; unthrottled a partition took ~0.4 s). Each variant: kill, retry once with the replica still down, bring it back, DROP PARTITION on the new table, copy again.

Partition Copy settings Failure Left behind Retry as is DROP PARTITION + copy
202605 (1,957,895) default blocks, default receive_timeout (300 s) Code 209 after 300 s 0 PASS PASS
202606 (1,894,737) 100,000-row blocks, receive_timeout = 30 Code 209 after 30 s 968,497 FAIL: 2,863,234 rows (1,894,737 distinct) PASS
202607 (1,957,895) as above + insert_deduplication_token Code 209 after 30 s 973,316 FAIL: 2,159,171 rows (1,952,523 distinct) PASS

Screens

ClickHouse's built-in /play on ch-s1r1 at the end of the run (state after B), captured with the notes site's scripts/capture-play.mjs:

Not verified here

Files

File What
run.py the lab: topology, data, prober, check, options A/B/D, failure case, selftest
report.py results/<version>.json → RESULTS.md, images/d-reads-<version>.svg
docker-compose.yml, conf.d/, users.d/, keeper/ the cluster
results/ JSON and transcript of each run
images/ the D timeline and the screens

ClickHouse OSS는 샤드를 재분배하지 않습니다. 문서의 "Rebalancing Data" 가이드가 첫 문장에서 그렇게 말하고, 수작업 방법 네 가지를 선호 순서대로 듭니다(2026-11-07, ClickHouse/ClickHouse e73957f의 docs/guides/oss/deployment-and-scaling/scaling-clusters.mdx). 이 실습은 로컬 클러스터(ClickHouse Keeper, 레플리카 2개인 샤드 2개, 그다음 세 번째 샤드)에서 그 방법들을 실행하고 샤드별 행 수, 걸린 시간, 이동 중 백그라운드 읽기 · 쓰기가 본 것, 파티션별 검증을 잽니다. 장애 사례로 복사 도중 원본 레플리카를 죽입니다.

검증: ClickHouse 26.8.36.3(LTS), 2026-11-07. 13:44 KST에 시작한 run.py all 한 번, 서버와 Keeper 모두 clickhouse/clickhouse-server:26.8.36.3 이미지, Docker Desktop VM(12 CPU · 7.7 GiB), 다른 컨테이너 없음. 다른 버전에서는 실행하지 않았습니다. 같은 날 앞선 실행은 results/earlier-runs/에 있습니다.

방법 옮기는 것 여기서 실행
A. 새 샤드 + <weight> 없음. 새 쓰기만 새 샤드로 기울임 예
B. remote()에서 INSERT … SELECT로 새 3샤드 배치에 모든 행, 다시 해시 예(remote, remoteSecure는 아님)
C. clickhouse-copier 모든 행 아니요 — 24.3에서 번들에서 빠졌고 "no longer supported"
D. FETCH PARTITION + ATTACH + DROP(스테이징으로 MOVE PARTITION TO TABLE) 파티션 단위 예

실행

Docker(컨테이너 7개에 약 3–4 GB 여유)와 Python 3만 있으면 됩니다. Python 패키지는 필요 없습니다.

cd workload/resharding
python3 run.py up          # Keeper + 서버 6대, HTTP 127.0.0.1:18201-18206
python3 run.py all         # selftest, A, D, 장애, B -> results/<version>.json · .log (약 17분, 그중 6분은 타임아웃 대기)
python3 report.py          # JSON에서 RESULTS.md와 images/d-reads-<version>.svg
python3 run.py down        # 컨테이너와 볼륨 삭제

python3 run.py a(또는 b, d, fail, selftest)는 새 2샤드 상태에서 방법 하나만 실행하고 results/<version>-<option>.json을 씁니다. python3 run.py resume b d는 기존 results/<version>.json에 이어서 실행합니다. ROWS=600000 python3 run.py all은 빠른 확인용입니다.

구성

결과

전체 표: RESULTS.md. 샤드별 행 수(샤드 1 / 2 / 3, 첫 레플리카의 로컬 count(), 프로브 행 포함):

방법 전 후 걸린 시간 검증
A. 가중치 1:1:2 + 새 행 300만 6,002,487 / 5,997,513 / 0 6,755,667 / 6,745,787 / 1,500,946 설정 반영 0.92–1.14초 PASS
B. remote() 복사, 파티션 6개 온라인 + 쓰기 동결 중 핫 파티션 6,002,487 / 5,997,513 / 0 4,017,368 / 3,973,248 / 4,014,084 일괄 2.6초, 쓰기 동결 0.31초 PASS
D. 옛 샤드마다 파티션 2개를 샤드 3으로 6,002,487 / 5,997,513 / 0 4,077,796 / 4,041,946 / 3,884,658 파티션마다 0.10–0.30초 PASS

D 동안의 읽기

장애: 복사 중 원본 레플리카 종료

복사가 파티션 절반을 읽었을 때 docker kill rs-ch-s2r2. 복사는 샤드마다 레플리카 2를 먼저 읽고 (remote('ch-s1r2|ch-s1r1,ch-s2r2|ch-s2r1', …), load_balancing = 'in_order') 레플리카 1에 쓰므로 죽인 노드는 원본일 뿐입니다. 복사는 초당 2 MB로 늦췄습니다(max_network_bandwidth, 제한 없이는 파티션 하나 약 0.4초). 변형마다 죽이기, 레플리카가 죽은 채로 한 번 재실행, 레플리카 복구, 새 테이블에서 DROP PARTITION, 다시 복사.

파티션 복사 설정 실패 남은 행 그대로 재실행 DROP PARTITION + 복사
202605(1,957,895) 기본 블록, 기본 receive_timeout(300초) 300초 뒤 Code 209 0 PASS PASS
202606(1,894,737) 블록 10만 행, receive_timeout = 30 30초 뒤 Code 209 968,497 FAIL: 2,863,234행(고유 1,894,737) PASS
202607(1,957,895) 위와 같음 + insert_deduplication_token 30초 뒤 Code 209 973,316 FAIL: 2,159,171행(고유 1,952,523) PASS

화면

실행이 끝난 뒤(B 이후 상태) ch-s1r1의 내장 /play, 노트 사이트의 scripts/capture-play.mjs로 캡처했습니다.

여기서 확인하지 않은 것

파일

파일 내용
run.py 실습 본체: 토폴로지, 데이터, 프로버, 검증, 방법 A/B/D, 장애 사례, selftest
report.py results/<version>.json → RESULTS.md, images/d-reads-<version>.svg
docker-compose.yml, conf.d/, users.d/, keeper/ 클러스터
results/ 실행별 JSON과 기록
images/ D 타임라인과 화면

Open this lab on GitHub →GitHub에서 이 실습 열기 →