Chapter 2: Write Path and Part Creation #

Welcome back! In Chapter 1: Data Part and Metadata, we met the main character of MergeTree: the data part. A data part is a self-contained directory with column files, marks, indexes, and checksums.

Now it’s time to answer a natural question: How does a data part get created?

This chapter walks through the Write Path: the journey every inserted row takes from INSERT to a visible, queryable data part.

The Use Case: Inserting and Reading Back #

Imagine you have a tiny weather table:

CREATE TABLE weather
(
    city       String,
    temp_c     Int32,
    event_date Date
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, city);

Now you run:

INSERT INTO weather (city, temp_c, event_date) VALUES
    ('Berlin', 21, '2026-01-01'),
    ('Paris',  24, '2026-01-02'),
    ('Berlin', 18, '2026-02-01'),
    ('Rome',   29, '2026-02-03');

ClickHouse answers:

Ok. 4 rows in set.

But what happened in between? How did those four rows become data parts that a SELECT can later read?

This chapter follows exactly that path.

Key Concept 1: MergeTreeSink Receives Chunks #

ClickHouse rarely processes one row at a time. It processes chunks: batches of rows split into columns.

The final object that receives these chunks for a MergeTree table is called MergeTreeSink.

Think of it as the loading dock of a printing press. A truck arrives with boxes of loose pages. Each box is a Chunk.

void MergeTreeSink::consume(Chunk & chunk)
{
    // Convert the chunk into a Block shaped like the table.
    Block block = getHeader().cloneWithColumns(chunk.detachColumns());

    // Now `block` contains rows from the INSERT query.
}

The MergeTreeSink does not write raw rows to disk. It prepares blocks and passes them to the next step.

Key Concept 2: MergeTreeDataWriter Splits by Partition #

The next important class is MergeTreeDataWriter.

Its job is to take one big block and split it into smaller blocks that belong to the same partition.

Why? Because a data part must belong to exactly one partition.

In our example, the original block contains two partitions:

• 202601 for January rows
• 202602 for February rows

After splitting, we get two BlockWithPartition objects.

This tiny structure is the bridge between a block and a partition:

struct BlockWithPartition
{
    std::shared_ptr<Block> block;
    MergeTreePartition partition;
    std::string partition_id;
};

The writer’s splitting method looks like this, simplified:

BlocksWithPartition parts = MergeTreeDataWriter::splitBlockIntoParts(
    std::move(block),
    max_parts_per_block,
    metadata_snapshot,
    context);

Now each BlockWithPartition can be written as its own temporary data part.

Key Concept 3: writeTempPart Creates a Temporary Part #

For each BlockWithPartition, the writer calls writeTempPart.

This creates a temporary part on disk. The directory name usually starts with tmp_insert_....

Why temporary? Because a part should not be visible until it is fully written and verified.

MergeTreeTemporaryPartPtr temp_part = writer.writeTempPart(
    block_with_partition,
    metadata_snapshot,
    context);

A MergeTreeTemporaryPart is a small wrapper around the real part.

struct MergeTreeTemporaryPart
{
    MergeTreeData::MutableDataPartPtr part;
    std::vector<Stream> streams;

    void finalize();
};

It also holds a temporary_directory_lock, which prevents background cleanup threads from deleting the temporary directory while it is still being written.

Tip: if you look at the data directory during an insert, you might catch a tmp_insert_... directory. That means the part is still in the printing stage.

Key Concept 4: Writing Columns, Marks, and Indexes #

Now comes the interesting part: how rows are serialized to disk.

The class responsible for writing actual bytes is MergeTreeDataPartWriterOnDisk and its subclasses.

For each column, it writes:

• a .bin file with the actual column data
• a .mrk3 file with marks, which are bookmarks for granules

A granule is a small group of rows. For example, one granule might contain 8192 rows.

struct Granule
{
    size_t start_row;       // first row in the granule
    size_t rows_to_write;   // number of rows in the granule
    size_t mark_number;     // which mark this granule maps to
    bool mark_on_start;     // should a mark be written at the beginning?
    bool is_complete;       // is the granule finished?
};

The writer calculates granules, writes column data, and records marks.

It also calculates the primary index and skip indices while writing. This is like printing the table of contents and page markers at the same time you print the pages.

void calculateAndSerializePrimaryIndex(const Block & index_block, const Granules & granules);
void calculateAndSerializeSkipIndices(const Block & skip_indexes_block, const Granules & granules);

For a Wide part, each column gets its own .bin and .mrk3 files. For a Compact part, all columns are packed into one data.bin. MergeTree chooses the layout automatically.

Key Concept 5: MergedBlockOutputStream Writes a Whole Part #

The class that coordinates the writing of one whole part is MergedBlockOutputStream.

Despite the complex name, its job is simple:

• take a sorted block
• hand it to the part writer
• write column data, marks, and indexes
• finalize part files and checksums

Why “Merged”? Because the same class is used for both:

• writing new inserted parts
• writing merged parts in the background

For inserts, rows are sorted before being written if needed. For merges, rows from several old parts are merged into one sorted stream.

class MergedBlockOutputStream final : public IMergedBlockOutputStream
{
public:
    void write(const Block & block) override;
    void writeWithPermutation(const Block & block, const IColumn::Permutation * permutation);
};

A permutation is just an array of row numbers that tells ClickHouse: “write row 5 first, then row 2, then row 9…” This lets the stream write rows in sorted order without keeping a second full block in memory.

Key Concept 6: Finalization Makes a Part Ready to Commit #

After all rows are written, the stream must finalize the part.

This step:

• writes the final marks
• finishes primary index and skip indices
• writes checksums.txt
• computes uncompressed sizes and hashes

The result is a Finalizer object. You call finish() to wait until all asynchronous writes are complete.

MergedBlockOutputStream::Finalizer finalizer = stream.finalizePartAsync(
    part,
    gathered_data,
    sync);

finalizer.finish();

After finalization, the temporary part is complete. But it is still not visible to queries.

Step-by-Step Walkthrough #

Let’s see the whole write path in one small diagram.

Step by step:

1. Client sends an INSERT query.
2. MergeTreeSink receives chunks of rows.
3. MergeTreeDataWriter splits the block by partition.
4. For each partition, a temporary part is written with .bin files, .mrk3 files, primary index, and checksums.
5. The temporary part is renamed to its final name.
6. The part is added to ActiveDataPartSet.
7. Ok. is returned to the client.
8. Now a SELECT can see the new rows.

For our weather insert, the simplified on-disk result might look like:

202601_1_1_0/   -- January rows
202602_2_2_0/   -- February rows

The exact block numbers depend on the table’s state, but the idea is the same: each partition has its own new data part.

Under the Hood: A Small Code Tour #

Let’s revisit a few key classes with simplified code.

MergeTreeSink #

This is the entry point for INSERT data.

void MergeTreeSink::consume(Chunk & chunk)
{
    Block block = getHeader().cloneWithColumns(chunk.detachColumns());

    auto blocks = MergeTreeDataWriter::splitBlockIntoParts(
        std::move(block),
        max_parts_per_block,
        metadata_snapshot,
        context);

    for (auto & block_with_partition : blocks)
        writeNewTempPart(block_with_partition);
}

The real implementation is more careful about memory and overlapping I/O, but this is the core idea.

MergeTreeDataWriter #

This class creates temporary parts.

MergeTreeTemporaryPartPtr MergeTreeDataWriter::writeTempPart(
    BlockWithPartition & block,
    StorageMetadataPtr metadata_snapshot,
    ContextPtr context)
{
    // 1. Create a temporary part with a name like tmp_insert_...
    // 2. Open a MergedBlockOutputStream for it
    // 3. Write the block
    // 4. Return the temporary part
}

It does not commit the part. Committing happens later, after the part is fully finalized.

MergeTreeDataPartWriterOnDisk #

This class is the real typographer. It knows how to serialize columns and compute indexes.

class MergeTreeDataPartWriterOnDisk : public IMergeTreeDataPartWriter
{
protected:
    virtual void addStreams(const NameAndTypePair & name_and_type,
                            const ASTPtr & effective_codec_desc) = 0;

    virtual ISerialization::SerializeBinaryBulkSettings getSerializationSettings() const = 0;
};

The wide-format writer is called MergeTreeDataPartWriterWide. It creates one stream per column.

MergedBlockOutputStream #

This class ties everything together.

class MergedBlockOutputStream final : public IMergedBlockOutputStream
{
public:
    void write(const Block & block) override;

    struct Finalizer
    {
        void finish();
    };

    Finalizer finalizePartAsync(...);
};

It writes blocks, finalizes marks, and computes checksums.

What About Background Merges? #

The write path we explored is for new inserts.

But MergeTree also has a background hero: the merge. A merge takes several small parts and writes one bigger, sorted part.

The interesting thing is: the merge path uses the same MergedBlockOutputStream. It reads rows from old parts, sorts or merges them, and writes a new part using the same mechanics.

That is why Chapter 5: Background Merges, Mutations, and Compaction Selection will feel familiar.

Summary #

Here is what we learned in this chapter:

• MergeTreeSink receives INSERT data as chunks.
• MergeTreeDataWriter splits blocks by partition.
• Each partition block becomes a temporary part.
• IMergeTreeDataPartWriter writes column data into .bin files and marks into .mrk3 files.
• It also calculates primary indexes and skip indexes.
• MergedBlockOutputStream coordinates writing one whole part.
• Finalization writes checksums and flushes all files.
• Only after the part is added to the active set does it become visible to SELECT.

The write path is like a printing press:

• loose pages arrive,
• they are sorted into chapters,
• the table of contents and page markers are printed,
• the pages are bound into a new book,
• and only then is the book placed on the library shelf.

Next, we will see how the library lets readers find books. In MergeTree terms: how a SELECT query finds the right parts and reads only the necessary granules.

Continue to Chapter 3: Read Path and Query Execution.

Happy querying!

Generated by AI Codebase Knowledge Builder