Korean keyword search for RAG — tokenizers, a morpheme column, hybrid retrieval

---

In a RAG stack on ClickHouse the vector half does not care about the language: Korean quality comes from the embedding model. The keyword half does care, and ClickHouse has no Korean analyser (nothing like nori or mecab-ko). This lab measures what each text-index tokenizer gets right and wrong when Korean questions are asked over Korean document chunks. It then measures what a morpheme column produced outside ClickHouse adds, and what hybrid retrieval adds on top.

Two labs cover the ground before this one: - local/releases/26.8 shows tokens() for asciiCJK, chinese and icu on one sentence. - usecase/fulltext-search measures recall lost to particles and spacing over 1M support tickets on Cloud.

This lab adds: - questions with labelled relevant chunks - a Kiwi morpheme column - splitByRegexp and sparseGrams - keyword ranking without BM25 - vector search and fusion - a Cloud check

Verified on ClickHouse 26.9.11.2 (clickhouse/clickhouse-server:26.9), 2026-10-06, steps 00–05 through tools/hol run and step 06 by hand. Kiwi: kiwipiepy 0.24.0, model 0.24.0. Step 06: Ollama 0.35.0, bge-m3 (digest 7907646426…), 1024 dimensions. Machine: Docker Desktop on macOS, 12 CPUs, 8 GB.

Results

40 questions, 300 chunks, ranked by matched query tokens ÷ query tokens (step 04). Index size is on the 300-chunk table (step 05). The body column itself is 27.6 KB compressed.

Keyword setup	recall@5	recall@10	MRR@10	unranked recall@10	index size	× body	scan without index (30k rows)
kiwi — morphemes, tokenizer = array	0.925	0.962	0.919	0.575	16.4 KB	0.60	6 ms
splitByRegexp — particle-stripping regexp	0.667	0.846	0.686	0.588	22.6 KB	0.82	794 ms
ngrams(2)	0.692	0.817	0.745	0.025	57.9 KB	2.10	15 ms
ngrams(3)	0.704	0.767	0.707	0.375	85.1 KB	3.09	16 ms
asciiCJK	0.562	0.700	0.613	0.025	29.1 KB	1.06	12 ms
sparseGrams	0.371	0.462	0.303	0.417	227.4 KB	8.24	52 ms
icu('ko')	0.346	0.450	0.359	0.338	26.4 KB	0.96	70 ms
splitByNonAlpha	0.296	0.396	0.306	0.308	26.4 KB	0.96	4 ms

Ranked recall@10 by failure case (8 questions each):

Keyword setup	particle 조사	spacing 띄어쓰기	ending 어미	mixed 한영	short 1음절
kiwi	1.000	1.000	0.812	1.000	1.000
splitByRegexp	1.000	0.500	0.854	0.875	1.000
ngrams(2)	0.812	0.875	0.750	0.896	0.750
ngrams(3)	0.812	0.750	0.500	0.958	0.812
asciiCJK	0.688	0.875	0.771	0.542	0.625
sparseGrams	0.750	0.375	0.104	0.521	0.562
icu('ko')	0.062	0.375	0.250	0.938	0.625
splitByNonAlpha	0.062	0.250	0.250	0.792	0.625

With the index, every tokenizer answers hasAllTokens(col, '법인카드 분실') at 30,000 rows in 2–3 ms and reads 800 rows; the scans in the first table read all 30,000. A plain LIKE scan took 3 ms at that size, so wall time does not separate the setups at this scale; rows read do. The 30k-row table repeats the 300 chunks 100 times, so dictionaries do not grow; it measures posting-list reads, not vocabulary. Three runs on the same day differed by under 10%.

Vector and hybrid (step 06, reciprocal rank fusion with k = 60 of the top 20 of each list):

Method	recall@5	recall@10	MRR@10
vector only (bge-m3)	0.979	1.000	0.975
hybrid vector + kiwi	0.962	1.000	0.966
hybrid vector + ngrams(2)	0.892	1.000	0.870
keyword kiwi	0.925	0.962	0.919
keyword ngrams(2)	0.692	0.817	0.745

Which keyword setup to use for Korean RAG, and why

1. A morpheme column indexed with tokenizer = array. Run a Korean analyser (here Kiwi) at ingest, store morphemes Array(String), search with the question's morphemes and rank by overlap. - It is best on every failure case but verb endings. It has the highest recall@5 (0.925) and the smallest index (0.6 × body). - The analyser runs outside ClickHouse, and nothing inside the server changes. - array is accepted on the Cloud service checked below. - It needs the same analyser, version and rule on the query side. lab_meta records them.
2. No analyser available: ngrams(2), ranked. - Recall@10 is 0.817 for 2.1 × the body on disk. - Never use it as a filter. hasAnyTokens returns 250 of 300 chunks for a question, so unranked recall@10 is 0.025. - The particle-stripping splitByRegexp gets 0.846 at recall@10 but only 0.5 on spacing. - That regexp also over-strips: 사이 becomes 사. - It costs the most to tokenize: 794 ms to scan 30k rows without the index. - It is not on the Cloud build checked below.
3. Avoid for question search: - icu('ko') and splitByNonAlpha keep the particle on the word: particle recall@10 is 0.062. - asciiCJK matches 296 of 300 chunks per question: no pruning, 38 of 38 granules. - sparseGrams: see the next section.
4. Always rank. Unranked filtering stays at 0.025–0.588 recall@10.
5. hasAllTokens on a question is not usable as is. - The question's endings (하나요, 어떻게) are not in the chunks, so every raw tokenizer returns nothing. - kiwi reaches 0.308 recall, because its rule drops those words.
6. On this corpus, hybrid did not beat vector search. - bge-m3 alone reaches recall@10 = 1.0. Hybrid with kiwi ties at @10 and is 0.017 lower at @5. - The two transliteration questions (아웃룩/Outlook, 팀즈/Teams) were found by keywords too, because the rest of the question carries them. - So this lab shows that the keyword setup decides how much a hybrid loses (vector + ngrams(2): 0.892 at @5). - It does not show what hybrid gains. - Hypothesis, unmeasured: the keyword half earns its place on corpora with many near-duplicate chunks, or exact identifiers the embedding model has not seen.

Things that surprised us (26.9.11.2)

• sparseGrams treats a String needle as a phrase.
• A String needle goes through compactTokens() (src/Functions/hasAnyAllTokens.cpp:64 at v26.9.11.2-stable).
• For sparseGrams that drops every gram contained in a longer one (src/Interpreters/ITokenizer.cpp:677).
• This is right for hasAllTokens. But hasAnyTokens(col, '<question>') then searches only the longest grams, so it behaves like a phrase search.
• For question 1, tokens() shares a gram with 97 chunks. The index returned 1 chunk, which matched only 나요?.
• Step 03 applies the same compaction, and its positive control (48 rows, all 8 tokenizers) fails the step without it.
• Workaround: pass tokens(question, 'sparseGrams') as an Array to get OR semantics. Reported as ClickHouse/ClickHouse#124156.
• sparseGrams also has a minimum length of 3, so tokens('결재', 'sparseGrams') is [].
• One text index per column. A second one fails with Code 36, Column b must not have more than one text index. Step 02 indexes MATERIALIZED copies of the body instead.
• The needle must be a constant.
• A column from a join is rejected with Code 44.
• A scalar subquery (SELECT question FROM queries WHERE qid = 1) is accepted, and the index is used.
• Steps 03–04 therefore compare token arrays, and step 03 checks them against the index.
• The 26.8 findings hold on 26.9. asciiCJK gives single syllables, and icu keeps particles. The locale (ko, ko_KR, en, ja) does not change the result.

Data

All of it is in the lab, so the T0 run needs neither Python nor network.

• data/chunks.jsonl: 300 chunks from 27 documents of a fictional company.
• Internal policy: remote work, leave, travel, corporate card, purchasing, approvals, security.
• IT helpdesk: VPN, Outlook, SSO, Teams, ERP error codes, devices.
• data/queries.jsonl: 40 questions phrased as an employee would ask a helpdesk bot.
• 8 per failure case, each with 2–3 relevant chunks (qrels) and a note naming its trap.
• particle: 결재선을 in the question, 결재선은 / 결재선이 in the chunks.
• spacing: 구매 요청서 against 구매요청서, and the other way round.
• ending: 잠겼어요 against 계정 잠금 / 잠긴 계정.
• mixed: lowercase codes, English glued to a particle (VPN이), error codes (ERP-4012, 0x80070005), and transliterations (아웃룩 against Outlook).
• short: one-syllable terms (키, 망, 폰, 앱, 차, 층, 팀, 표).
• Every question also has hard negatives: chunks that share its key term and answer something else.
• The text is synthetic, drafted for this lab with an AI model.
• The qrels are by construction: the relevant chunks were written for the question.
• Script checks: the trap holds in every relevant chunk, ids are contiguous, and there are 8 questions per case.
• A sample was read by hand.
• gen_data.py runs Kiwi once and writes 00-data.sql, which is committed.
• Rule: keep tags NNG NNP NR SL SN SH XR VV VA (the part before -), lowercase them, and drop 하 되 있 어떻.
• It is deliberately simple: 그렇 and 없 survive, for example.
• To regenerate: uv run usecase/korean-rag-tokenizers/gen_data.py. Re-running it with unchanged inputs writes a byte-identical file.

Steps

File	Tier	What it does
00-data.sql	T0	generated: chunks_src, queries, phrases, lab_meta
01-tokens.sql	T0	tokens() of 8 tokenizers on 11 failure-case phrases; the 26.8 re-check; sparseGrams minimum; constant needles
02-schema.sql	T0	chunks: the body, 7 MATERIALIZED copies each with its own text index (preprocessor = lower(...)), morphemes with tokenizer = array; index_granularity = 8 so pruning shows on 300 rows
03-recall.sql	T0	hasAny / hasAll filtering: recall, precision and chunks returned, by tokenizer and failure case; positive control against the index (48 rows, fails the step on any mismatch)
04-ranking.sql	T0	ranking by matched ÷ query tokens: recall@5/@10, MRR@10, unranked baseline, by failure case; keyword_ranked for step 06
05-index-cost.sql	T0	index bytes, distinct tokens, granules read (EXPLAIN indexes = 1), latency and rows read at 30k rows with and without the index
t1/06-hybrid.sh + t1/06-hybrid.sql	T1	aiEmbed through a local Ollama, a vector_similarity HNSW index, exact top 20, RRF with the kiwi and ngrams(2) lists
cloud/tokenizer-check.sql	by hand	which tokenizers a ClickHouse Cloud service accepts, as tokens() and as a text index; drops what it creates

t1/ and cloud/ sit outside the top-level NN-*.sql that tools/hol runs, so steps 00–05 stay T0.

Run it

python3 tools/hol run usecase/korean-rag-tokenizers

To read each step's output, keep the container and pipe the step in again:

python3 tools/hol run usecase/korean-rag-tokenizers --keep
docker exec -i hol-usecase-korean-rag-tokenizers clickhouse-client --multiline --multiquery < usecase/korean-rag-tokenizers/04-ranking.sql

Step 06 needs Ollama on the host, with the model pulled:

ollama pull bge-m3

Then run it against the kept container:

usecase/korean-rag-tokenizers/t1/06-hybrid.sh

What the script does: - It checks the container, the step-04 tables, Ollama and the model, and prints the Ollama version and model digest. - It gives the container's default user named_collection_control through a users.d file. The hol container has no right to CREATE NAMED COLLECTION without it. - It then runs 06-hybrid.sql. - The SQL reaches Ollama as host.docker.internal:11434. Docker Desktop provides that name; on Linux add --add-host=host.docker.internal:host-gateway (not tested).

python3 tools/hol down usecase/korean-rag-tokenizers

Two framework defaults that break Korean

Reproduced on 26.9.11.2 on 2026-10-06 in a scratch environment, with mock embeddings.

LlamaIndex — llama-index-vector-stores-clickhouse 0.8.0 (llama-index-core 0.14.25, clickhouse-connect 1.9.0).

How it searches keywords: - HYBRID and TEXT_SEARCH match keywords with regexes built in _build_text_search_statement / _build_hybrid_search_statement. - base.py:351 builds \b(?i)<token>\b for multiMatchAllIndices. - \b is an ASCII word boundary, so a Hangul keyword never matches: match('클릭하우스 테스트', '\\b(?i)클릭하우스\\b') = 0 on 26.9.11.2, while the English equivalent is 1. - No text index is created; the keyword path is a regex scan.

What goes wrong: - TEXT_SEARCH crashes. With no matching row, base.py:516 raises ValueError: tuple.index(x): x not in tuple. - HYBRID silently ignores the Korean keyword. - The second term is broken even for English. base.py:360 writes a single \b into a SQL literal, which ClickHouse reads as a backspace (hex('\b') = 08), so countMatches is always 0.

There is no constructor option. Override the two builders in a subclass:

from llama_index.vector_stores.clickhouse import ClickHouseVectorStore
from llama_index.vector_stores.clickhouse.base import _default_tokenizer, escape_str

class KoreanClickHouseVectorStore(ClickHouseVectorStore):
    def _terms(self, query_str):
        arr = "[" + ",".join("'" + escape_str(t) + "'" for t in _default_tokenizer(query_str)) + "]"
        hits = f"arraySum(arrayMap(p -> p > 0, multiSearchAllPositionsCaseInsensitiveUTF8(text, {arr})))"
        tf = f"log(1 + arraySum(arrayMap(t -> countSubstringsCaseInsensitiveUTF8(text, t), {arr})))"
        return hits, tf

    def _build_text_search_statement(self, query_str, similarity_top_k):
        hits, tf = self._terms(query_str)
        cols = ",".join(k for k in self._column_config if k != "vector")
        return (f"SELECT {cols}, score FROM {self._config.database}.{self._config.table} "
                f"WHERE score > 0 ORDER BY {hits} AS score DESC, {tf} AS d2 DESC LIMIT {similarity_top_k}")

    def _build_hybrid_search_statement(self, stage_one_sql, query_str, similarity_top_k):
        hits, tf = self._terms(query_str)
        cols = ",".join(k for k in self._column_config if k != "vector")
        return (f"SELECT {cols}, score FROM ({stage_one_sql}) tempt "
                f"ORDER BY {hits} AS d1 DESC, {tf} AS d2 DESC, score ASC LIMIT {similarity_top_k}")

How the override behaves: - Substring matching finds 클릭하우스 in 클릭하우스는 빠르다. - A query with no match still raises the ValueError above, so catch it around query(). - To use a text index instead, add WHERE hasAnyTokens(text, '<query>') on a column indexed as in this lab.

LangChain — langchain-community 0.4.2 (langchain_community/vectorstores/clickhouse.py).

The default index: - ClickhouseSettings.index_type defaults to "annoy" (line 78), with index_param = ["'L2Distance'", 100] (line 80). - ClickHouse removed annoy in 25.5, so on 26.9 the constructor fails with: Code: 80. DB::Exception: Unknown Index type 'annoy'. Available index types: hypothesis, text, vector_similarity, …

Override both settings:

from langchain_community.vectorstores import Clickhouse, ClickhouseSettings

# no ANN index (exact scan)
ClickhouseSettings(index_type=None, ...)
# or HNSW; the dimension goes inside index_param, and the metric must stay L2Distance
ClickhouseSettings(index_type="vector_similarity", index_param=["'hnsw'", "'L2Distance'", DIM], ...)

Pitfalls: - The metric must be L2Distance. The store's search is hard-coded to L2Distance (line 582); an index built on cosineDistance is not used. - Changing only index_type fails, with Vector similarity index must have three or six arguments. - index_query_params is broken. It emits SETTING instead of SETTINGS (line 576). The resulting syntax error is swallowed, and the search returns []. - langchain-community warns that it is being sunset. The langchain-clickhouse 0.1.0 on PyPI is a third-party package with no index options, not its successor.

ClickHouse Cloud

Checked on 2026-10-06 against a Cloud service on 26.6.1.2292, through mcp-clickhouse, by running cloud/tokenizer-check.sql statement by statement. The scratch database it creates was dropped afterwards.

Tokenizers: - system.tokenizers lists ngrams, splitByNonAlpha, sparseGrams, array, splitByString, asciiCJK, unicodeWord (and the bloom-filter names). It does not list icu, chinese, japanese or splitByRegexp. - tokens() works for the five listed in this lab. tokens(…, 'icu', 'ko') and tokens(…, 'splitByRegexp', …) fail with Code 42 on that build.

Text indexes, created exactly as 02-schema.sql declares them:

Tokenizer	Accepted on Cloud 26.6.1.2292
splitByNonAlpha, asciiCJK, ngrams(2), ngrams(3), sparseGrams (each with preprocessor = lower(b))	yes
array (the kiwi column)	yes
icu('ko')	no: Code: 36. Unknown tokenizer: 'icu': When validating secondary index
splitByRegexp(…, true)	no: Code: 36. Unknown tokenizer: 'splitByRegexp'

• One row inserted into the array and ngrams(2) tables was found by hasAnyTokens(m, ['요청서']) and hasAnyTokens(b, '구매 요청서').
• So the two setups recommended above, kiwi and ngrams(2), work on that service today.
• icu and splitByRegexp need a newer Cloud build.

Embeddings: - aiEmbed appears in system.functions there, but allow_experimental_ai_functions is 0. - The docs say AI functions are not available in Cloud services at the moment (AI functions, read 2026-10-06). - On Cloud the application produces the embeddings and inserts them as Array(Float32). The vector_similarity part of step 06 is unchanged.

Other notes: - The docs mark the japanese tokenizer's dictionary configuration as not supported on Cloud (text index, read 2026-10-06). - To repeat the check on another service, run cloud/tokenizer-check.sql statement by statement. It drops what it creates. Ask the service owner first.

Not covered

• BM25. It is not in a release: ClickHouse/ClickHouse#117519, a pull request, was open on 2026-10-06.
• A Korean analyser inside ClickHouse.
• Scale. 300 chunks are enough to separate tokenizers by recall, but not to measure latency or hybrid gains. Both are open questions for a larger corpus.

Author

Ken (ClickHouse Solution Architect) · 2026-10-06

---

python3 tools/hol run usecase/korean-rag-tokenizers

python3 tools/hol run usecase/korean-rag-tokenizers --keep
docker exec -i hol-usecase-korean-rag-tokenizers clickhouse-client --multiline --multiquery < usecase/korean-rag-tokenizers/04-ranking.sql

ollama pull bge-m3

usecase/korean-rag-tokenizers/t1/06-hybrid.sh

python3 tools/hol down usecase/korean-rag-tokenizers