Chapter 1: Data Part and Metadata #

Welcome! If you are new to ClickHouse and MergeTree, you have come to the right place. This chapter is a friendly, no-fear introduction to two core ideas: data parts and metadata.

Imagine a paper notebook. The whole notebook is your MergeTree table. Each chapter is a data part. Each chapter knows which pages it covers, and marks are little bookmarks that help you jump to the right page. This chapter is about those chapters and the table of contents that keeps track of them.

Let’s start with a simple goal.

The Use Case: What Happens When You Insert Rows? #

Suppose you run this query:

INSERT INTO weather (city, temp_c) VALUES ('Berlin', 21), ('Paris', 24);

You want ClickHouse to save these two rows so you can read them later. But ClickHouse does not store rows one by one in a giant file. Instead, it packs rows into a data part.

After the insert, something like this exists on disk:

2026-01-01_1_1_0/
├── columns.txt      -- column names and types
├── checksums.txt    -- hashes for every file
├── count.txt        -- number of rows
├── data.bin         -- actual column data
└── data.mrk3        -- marks, the "bookmarks"

That small directory is a data part. It is self-contained: it knows its columns, rows, checksums, indexes, and partition. If you want to read those rows later, ClickHouse opens this directory and reads it.

Now let’s unpack each piece.

Key Concept 1: A Data Part Is a Mini Table #

A data part is not just a random file. It is a complete, independent unit of a table. It stores rows for one range of block numbers. Block numbers are just increasing insert sequence numbers. The first insert might get block number 1, the next block number 2, and so on.

Why do this? Because it makes background operations easy. MergeTree can merge two parts into a bigger part, delete old parts, and never mix up data. Each part is like a chapter that does not depend on other chapters.

The name of the part is itself metadata. For example:

2026-01-01_1_1_0

means:

• partition: 2026-01-01
• min block: 1
• max block: 1
• merge level: 0

In C++ code, this is represented by a structure called MergeTreePartInfo.

struct MergeTreePartInfo
{
    String partition_id;   // which partition this part belongs to
    Int64 min_block;       // first block number in this part
    Int64 max_block;       // last block number in this part
    UInt32 level;          // merge generation (0 = fresh insert)
    Int64 mutation;        // used after mutations
};

Don’t worry about mutation yet. Just remember that this structure describes the range of data inside the part. Two parts are disjoint if their block ranges do not overlap. One part can contain another part if it has a wider block range and a higher level.

Key Concept 2: Marks Are Bookmarks #

Inside every part, there are marks. Marks tell ClickHouse where each granule starts. A granule is a small chunk of rows, usually between 8192 and 1 million rows depending on settings.

Think of marks like page numbers in a book. If you want to find WHERE city = 'Berlin', ClickHouse uses an index to skip to the right mark, then reads only that granule. Marks are stored in files like data.mrk3.

We will talk much more about indexes and marks in Chapter 4: Indexes and Conditions. For now, just know that every data part carries its own marks.

Key Concept 3: Every Part Has a Lifecycle #

A part is not visible to queries the moment it is created. It has to go through states. These states are like the life of a person: being born, becoming visible, growing old, and finally being removed.

Here is the enum from MergeTreeDataPartState.h:

enum class MergeTreeDataPartState
{
    Temporary,       // being written, not visible yet
    PreActive,       // stored in the table, but not visible
    Active,          // visible to SELECT queries
    Outdated,        // replaced by a bigger part, still used by old queries
    Deleting,        // being removed by a background cleaner
    DeleteOnDestroy, // moved away; will be deleted later
};

The typical journey is:

Temporary -> PreActive -> Active -> Outdated -> Deleting

Why so many steps? Because ClickHouse needs to be safe. A part should only become visible after all its files are written correctly. And when a part is replaced by a merge, old queries that already started reading it should be allowed to finish. That is why the part becomes Outdated before it is finally deleted.

Key Concept 4: Wide, Compact, and Packed Parts #

Data parts come in different physical layouts. The main ones are Wide and Compact. There is also a special Packed storage type.

• Wide parts: each column is stored in its own file. Great for large parts.
• Compact parts: all columns are stored in one data.bin file. Great for small parts.
• Packed parts: all files are stored inside one archive file called data.packed. Useful for cloud storage.

The C++ types look like this:

enum class MergeTreeDataPartType
{
    Wide,
    Compact,
    Unknown,
};

enum class MergeTreeDataPartStorageType
{
    Full,
    Packed,
    Unknown,
};

ClickHouse chooses the format automatically. Small inserts usually become Compact parts; large merges usually become Wide parts. That decision is made by the part builder.

Key Concept 5: ActiveDataPartSet Keeps the Table of Contents #

A MergeTree table has many data parts. Some are active, some are outdated. The ActiveDataPartSet is the table of contents that tracks which parts are currently visible.

It needs to answer questions like:

• Is this new part already covered by a bigger part?
• Which parts overlap?
• Which parts should be replaced?

Here is a tiny version of its API:

class ActiveDataPartSet
{
public:
    enum class AddPartOutcome
    {
        Added,
        HasCovering,
        HasIntersectingPart,
    };

    bool add(const MergeTreePartInfo & part_info, Strings * out_replaced_parts);
    String getContainingPart(const MergeTreePartInfo & part_info) const;
};

When you add a new part, the set may remove parts that are fully covered by it. For example:

Before: all_1_10_0, all_11_20_0
Add:    all_1_20_1
After:  all_1_20_1

The old two parts become Outdated. This is exactly how merges work: merge several small parts, then add the merged part to the active set. The small parts are no longer active.

Solving the Use Case: Inserting Rows #

Now we can walk through our original insert example with more detail.

1. The client sends an insert.
2. MergeTree assigns the next block number, say 42.
3. The part builder creates a part named something like 2026-01-01_42_42_0.
4. The part storage writes files to disk.
5. The part becomes PreActive, then Active after it is added to ActiveDataPartSet.
6. The client sees Ok. and the rows are now queryable.

Example output:

Input:  INSERT INTO weather VALUES ('Berlin', 21), ('Paris', 24);
Output: Ok. 2 rows in set.

What happened internally? A data part directory was created containing column files, marks, and checksums. That directory is now one “chapter” in the table’s book.

Under the Hood: How the Part Builder Works #

The MergeTreeDataPartBuilder creates the correct type of part. It is like a factory that decides whether your part should be Wide or Compact.

Here is a simplified version of its build() method:

std::shared_ptr<IMergeTreeDataPart> MergeTreeDataPartBuilder::build()
{
    switch (part_type)
    {
        case PartType::Wide:
            return std::make_shared<MergeTreeDataPartWide>(...);
        case PartType::Compact:
            return std::make_shared<MergeTreeDataPartCompact>(...);
        default:
            throw Exception("Unknown part type");
    }
}

The builder also checks the disk to see what files already exist. For example, DataPartStorageOnDiskFull wraps a directory on a disk:

bool DataPartStorageOnDiskFull::exists() const
{
    return volume->getDisk()->existsDirectory(fs::path(root_path) / part_dir);
}

This is how ClickHouse asks “does this part already exist?” before creating a new one.

Under the Hood: Checksums Keep Parts Honest #

Every part contains a checksums.txt file. This file stores a hash and size for every file in the part. It is a bit like a packing list with a fingerprint for each item.

In code, checksums are represented by MergeTreeDataPartChecksums:

struct MergeTreeDataPartChecksums
{
    std::map<String, Checksum> files;

    void checkEqual(const MergeTreeDataPartChecksums & rhs,
                    bool have_uncompressed,
                    const String & part_name) const;
};

When ClickHouse reads a part, it compares the actual files with the checksums. If something changed on disk, the table can notice the problem. This is a safety net for data integrity.

Putting It All Together #

Let’s summarize the important ideas:

1. A data part is a self-contained directory of files: columns, marks, indexes, checksums, and partition metadata.
2. MergeTreePartInfo describes the part’s partition, block range, level, and mutation version.
3. A part moves through states: Temporary → PreActive → Active → Outdated → Deleting.
4. Parts can be Wide, Compact, or Packed.
5. ActiveDataPartSet is the table of contents that tracks active parts and their relationships.
6. Checksums protect every file inside a part.

Now when you insert rows into a MergeTree table, you know the real story: ClickHouse is creating a new data part, giving it metadata, checking it, and adding it to the active set.

What’s Next? #

You have met the main character of MergeTree: the data part. Now it’s time to see how it is created in detail. The next chapter walks through the insert and merge paths, showing how part builders, disk transactions, and background tasks work together.

Continue to Chapter 2: Write Path and Part Creation.

Happy querying!

Generated by AI Codebase Knowledge Builder