Chapter 3: Read Path and Query Execution #

Welcome back! In Chapter 2: Write Path and Part Creation, we followed inserted rows as they became new data parts. Now let’s turn the page and look at the other side of the story: how does ClickHouse read data back out?

The answer is the Read Path. This is the machinery that turns a SELECT query into a set of small, parallel read jobs.

The Use Case: Reading Only What You Need #

Imagine our little weather table again:

CREATE TABLE weather
(
    city       String,
    temp_c     Int32,
    event_date Date
)
ENGINE = MergeTree
ORDER BY (event_date, city);

You ask:

SELECT city, temp_c
FROM weather
WHERE event_date = '2026-01-01'
  AND city = 'Berlin';

ClickHouse must answer with:

Berlin  21

But it should not read every file from every data part. It should only read the parts and marks that could contain Berlin on 2026-01-01.

How does that work? Let’s use an analogy.

The Warehouse Analogy #

Imagine a giant warehouse full of books. Each data part is a bookshelf. Each shelf has marks — little bookmarks on every page range.

Now a customer asks:

“Please find every page where the city is Berlin and the date is January 1st.”

You don’t send a helper to every shelf to read every page. Instead:

1. The index analysis looks at the card catalog and chooses only the shelves that might match.
2. A dispatcher gives each helper a cart.
3. Each cart has a shelf number and a page range.
4. The helper walks to that shelf, opens only those pages, and writes down matching rows.

In MergeTree terms:

Let’s meet each character.

Key Concept 1: RangesInDataPart Is the Cart #

After index analysis, ClickHouse doesn’t say “read this entire part.” It says:

“Read only these mark ranges from this part.”

That information is stored in a structure called RangesInDataPart.

Here is a simplified version:

struct RangesInDataPart
{
    DataPartPtr data_part;            // which bookshelf
    MarkRanges ranges;                // which bookmarks to read
    size_t part_index_in_query;       // helper for _part_index
    RangesInDataPartReadHints read_hints;
};

MarkRanges is just a list of mark ranges:

struct MarkRange
{
    size_t begin;
    size_t end;

    size_t getNumberOfMarks() const
    {
        return end - begin;
    }
};

A mark range means: start at mark begin, read until mark end. Think of it as “start at bookmark 5 and keep reading until bookmark 8.”

Key Concept 2: The Read Pool Is the Dispatcher #

A part can be huge. Reading it with one worker would be slow. MergeTree wants to split work among many threads.

That’s the job of the read pool.

The default pool is MergeTreeReadPool. It takes all the RangesInDataPart objects and cuts them into smaller pieces.

class MergeTreeReadPool
{
public:
    MergeTreeReadTaskPtr getTask(
        size_t task_idx,
        MergeTreeReadTask * previous_task);
};

Each worker thread calls getTask() when it is ready for more work. The pool decides which part and which mark range to give next.

There is also a special pool called MergeTreeReadPoolInOrder. It does the same thing, but it preserves the order of ranges. This is useful when a query wants rows in sorting-key order, for example when you use ORDER BY on the primary key.

class MergeTreeReadPoolInOrder : public MergeTreeReadPoolBase
{
public:
    bool preservesOrderOfRanges() const override
    {
        return true;
    }
};

So:

• MergeTreeReadPool = fast, parallel, maybe out of order.
• MergeTreeReadPoolInOrder = still parallel, but keeps ranges in order.

The dispatcher decides which pool to use based on the query.

Key Concept 3: MergeTreeReadTask Is One Work Order #

When the pool gives work to a reader, the work order is a MergeTreeReadTask.

A task says:

“Read these mark ranges from this part.”

It also contains everything needed to actually read the data.

Here is a very simplified version:

struct MergeTreeReadTask
{
    MergeTreeReadTaskInfoPtr info;      // part, columns, settings
    MarkRanges mark_ranges;             // which marks to read
    Readers readers;                    // main + prewhere readers
    MergeTreeReadersChain readers_chain; // chain of range readers
};

Do not worry about every field. Focus on the idea: one task = one unit of work for one reader.

Key Concept 4: MergeTreeSelectProcessor Is the Worker #

Each thread runs a MergeTreeSelectProcessor. It keeps asking the pool for tasks and then reads them.

A simplified read loop looks like this:

ChunkAndProgress MergeTreeSelectProcessor::read()
{
    // Ask the pool for a task if we don't have one.
    if (!task || task->isFinished())
        task = pool->getTask(thread_id, previous_task);

    if (!task)
        return {};   // no more work

    return task->read();
}

The processor is stateful. It remembers which task it is currently working on. When the task is finished, it fetches the next one.

Key Concept 5: The Range Reader Walks the Shelf #

Inside a task, actual reading is done by IMergeTreeReader.

The name “range reader” is perfect: it reads the data between pairs of marks in the same part.

class IMergeTreeReader
{
public:
    virtual size_t readRows(
        size_t from_mark,
        bool continue_reading,
        size_t max_rows_to_read,
        MutableColumns & res_columns) = 0;
};

There is also an even lower-level class called MergeTreeReaderStream. It knows how to open a column file, seek to a mark, and read compressed data.

class MergeTreeReaderStream
{
public:
    void seekToMark(size_t row_index);
    ReadBuffer * getDataBuffer();
};

But readers are usually used together in a chain.

Key Concept 6: MergeTreeReadersChain Connects the Readers #

A query often needs several columns. Some columns are read for WHERE, some for PREWHERE, and some may be transformed by on-the-fly mutations.

The MergeTreeReadersChain orchestrates all of them.

class MergeTreeReadersChain
{
public:
    using ReadResult = MergeTreeRangeReader::ReadResult;

    ReadResult read(
        size_t max_rows,
        MarkRanges & ranges,
        std::vector<MarkRanges> & patch_ranges);
};

Think of the chain as a conveyor belt. Data moves through:

1. Main WHERE reader
2. PREWHERE readers
3. Mutation steps
4. Final column materialization

For example, PREWHERE is an optimization. It reads only the columns used in the PREWHERE clause first. If a row does not pass the condition, the chain never reads the remaining expensive columns for that row.

The chain also applies on-the-fly mutations. If a part has not yet been mutated on disk, the mutation can be applied while reading. We will talk more about mutations in Chapter 5: Background Merges, Mutations, and Compaction Selection. For now, just know that the read path can “repair” data on the way out.

Putting It All Together: A SELECT Query #

Let’s walk through a full example.

Our query:

SELECT city, temp_c
FROM weather
WHERE event_date = '2026-01-01'
  AND city = 'Berlin';

Suppose the table has these parts:

202601_1_1_0/   -- January rows
202602_2_2_0/   -- February rows

Step 1: Index Analysis Narrows the Parts #

Index analysis checks the primary key and removes parts that cannot match.

• 202602_2_2_0 is February, so it is skipped.
• 202601_1_1_0 is January, so it remains.

The result is a RangesInDataPart:

RangesInDataPart
{
    data_part = 202601_1_1_0,
    ranges = { {begin=0, end=3} }
}

Step 2: The Read Pool Prepares Tasks #

The pool cuts the ranges into small tasks. With one thread, it might return one task for the whole part. With four threads, it might split the marks into four smaller tasks.

Step 3: A Select Processor Reads a Task #

For each task:

1. Create a reader for the part.
2. Read rows from the mark ranges.
3. Apply PREWHERE filtering if needed.
4. Apply missing default values, column conversions, and mutations.
5. Return a chunk of rows.

Step 4: The Result Is Returned #

The query gets back the matching rows:

Berlin  21

No February part was opened. No unnecessary pages were read.

A Diagram of the Read Path #

Here is the whole journey in one picture:

The read pool keeps giving out new tasks until all selected parts are fully read.

Under the Hood: A Small Code Tour #

Let’s look at some real code from the project, simplified for learning.

MergeTreeSelectProcessor.h #

The processor stores the pool and the current task:

class MergeTreeSelectProcessor
{
private:
    const MergeTreeReadPoolPtr pool;
    MergeTreeReadTaskPtr task;
    Block result_header;
};

The read() method is the main entry point:

ChunkAndProgress MergeTreeSelectProcessor::read()
{
    if (!task || task->isFinished())
        task = pool->getTask(index, previous_task);

    if (!task)
        return {};

    return task->read();
}

MergeTreeReadTask.h #

A task combines the part info, mark ranges, and readers:

struct MergeTreeReadTask
{
    MergeTreeReadTaskInfoPtr info;
    MarkRanges mark_ranges;
    Readers readers;
    MergeTreeReadersChain readers_chain;

    BlockAndProgress read();
};

IMergeTreeReader.h #

The reader reads rows between marks:

class IMergeTreeReader
{
public:
    virtual size_t readRows(
        size_t from_mark,
        bool continue_reading,
        size_t max_rows_to_read,
        MutableColumns & res_columns) = 0;
};

MergeTreeReadersChain.h #

The chain coordinates all readers:

class MergeTreeReadersChain
{
public:
    ReadResult read(
        size_t max_rows,
        MarkRanges & ranges,
        std::vector<MarkRanges> & patch_ranges);
};

Notice how small and focused each class is. The read path is a pipeline of small responsibilities.

Why Is This Design Nice? #

The read path is designed for parallelism and efficiency.

Parallelism comes from tasks. If you have 100 parts and 8 threads, the read pool can split marks into 8 or more tasks. Each thread reads its own ranges independently.

Efficiency comes from ranges. Because marks point to compressed blocks, MergeTree can skip enormous amounts of data. It reads only the granule boundaries that might contain matches.

Imagine reading a 1000-page book by only opening 5 specific page ranges. That is the whole idea.

Summary #

In this chapter, you learned:

• A SELECT query does not scan all parts blindly.
• Index analysis produces RangesInDataPart objects, which are like carts with shelf ranges.
• MergeTreeReadPool and MergeTreeReadPoolInOrder cut those ranges into small MergeTreeReadTasks.
• MergeTreeSelectProcessor drives each task.
• IMergeTreeReader reads rows between marks.
• MergeTreeReadersChain connects multiple readers and handles PREWHERE, default values, column conversions, and on-the-fly mutations.

The read path is the reason a query over terabytes of data can still return quickly when indexes and marks do their job.

But we still haven’t looked closely at how index analysis decides which parts and marks to keep. That is the secret that makes the read path fast.

Continue to Chapter 4: Indexes and Conditions.

Happy querying!

Generated by AI Codebase Knowledge Builder