ClickHouse MergeTree Internals on Random Tech Excursions

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 1: Data Part and Metadata #

Welcome! If you are new to ClickHouse and MergeTree, you have come to the right place. This chapter is a friendly, no-fear introduction to two core ideas: data parts and metadata.

Imagine a paper notebook. The whole notebook is your MergeTree table. Each chapter is a data part. Each chapter knows which pages it covers, and marks are little bookmarks that help you jump to the right page. This chapter is about those chapters and the table of contents that keeps track of them.

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 2: Write Path and Part Creation #

Welcome back! In Chapter 1: Data Part and Metadata, we met the main character of MergeTree: the data part. A data part is a self-contained directory with column files, marks, indexes, and checksums.

Now it’s time to answer a natural question: How does a data part get created?

This chapter walks through the Write Path: the journey every inserted row takes from INSERT to a visible, queryable data part.

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 3: Read Path and Query Execution #

Welcome back! In Chapter 2: Write Path and Part Creation, we followed inserted rows as they became new data parts. Now let’s turn the page and look at the other side of the story: how does ClickHouse read data back out?

The answer is the Read Path. This is the machinery that turns a SELECT query into a set of small, parallel read jobs.

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 4: Indexes and Conditions #

Welcome back! In Chapter 3: Read Path and Query Execution, you followed a SELECT query as it created tasks and read mark ranges. But one mystery remained: how does ClickHouse decide which mark ranges to read in the first place?

The answer is Indexes and Conditions.

Imagine a reference book with two helpful features. First, a table of contents tells you which chapter to open. Second, pages have highlighted sentences, so you can quickly skip entire pages if they don’t contain what you need.

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 5: Background Merges, Mutations, and Compaction Selection #

Welcome back! In Chapter 4: Indexes and Conditions, you saw how SELECT queries use indexes to skip data. But data does not stay still. New inserts keep arriving, old rows need corrections, and too many tiny data parts can make reads slow.

This chapter is about housekeeping. MergeTree has background jobs that tidy up the table. They merge small parts into bigger parts and apply mutations. There is also a selection policy that decides which parts should be merged first.

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 6: Replication and ZooKeeper Coordination #

Welcome back! In Chapter 5: Background Merges, Mutations, and Compaction Selection, you saw how a single MergeTree table tidies itself up with merges and mutations. But what if the table lives on several machines? What if one machine is offline? How do all the machines agree on which data parts exist?

That is the job of Replication and ZooKeeper Coordination.