Chapter 5: Background Merges, Mutations, and Compaction Selection #

Welcome back! In Chapter 4: Indexes and Conditions, you saw how SELECT queries use indexes to skip data. But data does not stay still. New inserts keep arriving, old rows need corrections, and too many tiny data parts can make reads slow.

This chapter is about housekeeping. MergeTree has background jobs that tidy up the table. They merge small parts into bigger parts and apply mutations. There is also a selection policy that decides which parts should be merged first.

Imagine a library. Every insert creates a thin booklet. After a while, there are hundreds of thin booklets. Reading a book that is split into hundreds of booklets is annoying. A librarian combines them into bigger books. Sometimes a page needs a correction, so the librarian rewrites that page. This chapter is about that librarian: how it chooses booklets to combine, how it applies corrections, and how it tracks the work in progress.

The Use Case: Many Tiny Parts and a Correction #

Let’s go back to our weather table:

CREATE TABLE weather
(
    city       String,
    temp_c     Int32,
    event_date Date
)
ENGINE = MergeTree
ORDER BY (event_date, city);

Now imagine you run many small INSERT queries:

INSERT INTO weather VALUES ('Berlin', 21, '2026-01-01');

After 100 inserts, there are 100 small data parts. A SELECT that reads January data must read all 100 parts. That works, but it would be much faster to read one big part.

You also discover that the temperatures were accidentally stored in Fahrenheit. You need to correct them:

ALTER TABLE weather
    UPDATE temp_c = round((temp_c - 32) * 5 / 9)
    WHERE event_date < '2026-03-01';

This is a mutation. MergeTree will rewrite the affected parts in the background.

The rest of this chapter explains how both of these things happen.

Key Concept 1: Background Tasks Are Coroutines #

Before looking at merges, let’s look at the smallest unit of background work: the task.

A background task is a self-made coroutine. That sounds complex, but it is simple: the task has an executeStep() method. Every time you call it, the task does one little piece of work. If it returns true, it wants to run again. If it returns false, it is finished.

This is the interface from IExecutableTask.h:

class IExecutableTask
{
public:
    virtual bool executeStep() = 0; // true = one more step
    virtual void onCompleted() = 0; // called at the end
    virtual void cancel() noexcept = 0;
};

The background pool keeps calling executeStep() until the task says false. This is how a huge merge can run without blocking the whole server. It gives back control between steps.

Key Concept 2: MergeTask and MutateTask Do the Heavy Lifting #

A merge is not one giant operation. It is a MergeTask. A mutation is also not one giant operation. It is a MutateTask.

Both tasks look similar from the outside. For example, here is a simplified MutateTask:

class MutateTask
{
public:
    bool execute();                    // do one step
    void cancel() noexcept;
    std::future<MergeTreeData::MutableDataPartPtr> getFuture();
};

You call execute() again and again. When it is done, getFuture() gives you the new data part.

MergeTask works the same way. The important idea is that MergeTreeDataMergerMutator builds these tasks, and smaller wrapper tasks feed them to the background pool.

Key Concept 3: MergeTreeDataMergerMutator Is the Librarian #

The main helper for background merges and mutations is MergeTreeDataMergerMutator. It knows how to create the right task for a pile of parts.

Here is a simplified view of its interface:

class MergeTreeDataMergerMutator
{
public:
    MergeTaskPtr mergePartsToTemporaryPart(...);   // build merge task
    MutateTaskPtr mutatePartToTemporaryPart(...);  // build mutate task
};

You give it a set of old parts, and it returns a task that will write a new temporary part. Later, the new part is renamed and made active.

The important word is temporary. The new part is written as tmp_merge_... or tmp_mut_.... It becomes visible only after it is fully written and committed. This is the same idea you saw in Chapter 2: Write Path and Part Creation: a part is not active until it is safely complete.

Key Concept 4: MergeList Shows What Is Running #

When a merge or mutation runs, it appears in system.merges. The MergeList class keeps track of all running background operations.

A simplified version of the information stored for each operation looks like this:

struct MergeInfo
{
    String result_part_name;
    Array source_part_names;
    Float64 progress;
    UInt64 rows_read;
    UInt64 rows_written;
};

You can watch merges while they run:

SELECT database, table, result_part_name, progress
FROM system.merges;

The output shows a row for each running merge or mutation. This is like a sign on the library wall: “Currently combining booklets 1–10 into one book, 42% done.”

Key Concept 5: Compaction Selection Decides What to Merge #

Merging is expensive. You should not merge everything at once. MergeTree needs a policy to decide which piles of parts to combine first.

This is called compaction selection. It has three parts:

1. IPartsCollector collects active parts into ranges.
2. MergeSelectorApplier uses a merge selector, like SimpleMergeSelector or a TTL selector, to choose ranges.
3. Merge predicates check whether the chosen merge is safe.

Let’s look at each part.

IPartsCollector: The Inventory #

The collector walks through active data parts and groups them into ranges. A range is just an ordered list of parts that could be merged into one part.

Here is the simplified interface:

class IPartsCollector
{
public:
    virtual CollectedPartsRanges grabAllPossibleRanges(...) const = 0;
    virtual std::expected<PartsRange, PreformattedMessage>
        grabAllPartsInsidePartition(...) const = 0;
};

The first method is used for normal background merges. The second is used when you run OPTIMIZE TABLE ... FINAL or ALTER TABLE ... MODIFY PARTITION.

MergeSelectorApplier: The Policy #

Once the collector returns ranges, the MergeSelectorApplier asks a selector which ranges are the best to merge.

Here is a simplified version:

class MergeSelectorApplier
{
public:
    MergeSelectorChoices chooseMergesFrom(
        const PartsRanges & ranges,
        const PartitionsStatistics & partitions_stats,
        const IMergePredicate & predicate,
        ...) const;
};

The output is a list of MergeSelectorChoice objects. Each choice says which parts to merge and what kind of merge to do:

struct MergeSelectorChoice
{
    PartsRange range;        // parts to merge
    MergeType merge_type{};  // regular or TTL
    bool final = false;
};

SimpleMergeSelector: Balancing Work and Read Speed #

The default selector is SimpleMergeSelector. It tries to balance two opposite goals:

• Keep the number of parts low so reads are fast.
• Avoid too many merges so writes do not become too expensive.

The main knob is called base. A small base means merges happen more eagerly. A large base means fewer, wider merges.

Here is a very simplified version of its settings:

class SimpleMergeSelector final : public IMergeSelector
{
public:
    struct Settings { size_t max_parts_to_merge_at_once = 100; double base = 5; };
    PartsRanges select(...) const override;
};

So base = 5 is a typical starting point. The selector uses heuristics involving part size, part age, and total part count to adjust the behavior automatically.

TTL Selectors: Cleaning Up Expired Data #

There are also special TTL selectors. They select parts that contain expired rows or parts that need recompression.

class TTLRowDeleteMergeSelector : public ITTLMergeSelector
{
    time_t getTTLForPart(const PartProperties & part) const override;
    bool canConsiderPart(const PartProperties & part) const override;
};

These selectors are used when a table has TTL rules, such as “delete rows older than 90 days.”

Merge Predicates: The Safety Check #

Before any merge is accepted, a predicate asks a simple question: is it safe to merge these parts?

The MergeTreeMergePredicate checks things like:

• Are these parts still active?
• Is another operation already trying to merge one of them?
• Could a new insert appear between them?
• Are there any ALTER commands that must be applied first?

Here is a simplified interface:

class MergeTreeMergePredicate final : public IMergePredicate
{
public:
    std::expected<void, PreformattedMessage> canMergeParts(
        const PartProperties & left,
        const PartProperties & right) const override;
};

The name std::expected<void, PreformattedMessage> is a fancy way of saying: this method returns either “okay” or “error message”.

Solving the Use Case: What Happens Step-by-Step #

Let’s put the pieces together for our weather table.

After 100 small inserts, the table has 100 active parts. A background scheduler wakes up and asks:

1. Collector: “What active parts exist?”

   • It returns parts grouped into ranges.

2. Selector: “Which range should we merge?”

   • SimpleMergeSelector chooses 50 small parts in the same partition.

3. Predicate: “Is this merge safe?”

   • The predicate says yes.

4. Task builder: MergeTreeDataMergerMutator creates a MergeTask.

5. Background pool: calls executeStep() on the wrapper task until the merge finishes.

6. Commit: the new big part replaces the 50 small parts. They become outdated and are later deleted.

Here is a simple diagram of the selection part:

And here is a diagram of the execution part:

For the mutation example, the same flow happens, but MergeSelectorApplier is not needed. A mutation command knows exactly which parts it must rewrite. The background pool runs a MutateTask for each affected part.

Under the Hood: Code Walkthrough #

Now let’s look at the actual wrapper tasks that connect IExecutableTask with MergeTask and MutateTask.

MergePlainMergeTreeTask #

The class MergePlainMergeTreeTask wraps a MergeTask. It uses a small state machine:

• NEED_PREPARE: build the MergeTask.
• NEED_EXECUTE: call merge_task->execute().
• NEED_FINISH: commit the new part and clean up.

Here is a simplified version of executeStep():

bool MergePlainMergeTreeTask::executeStep()
{
    if (state == State::NEED_PREPARE) { prepare(); state = State::NEED_EXECUTE; return true; }
    if (state == State::NEED_EXECUTE)
        if (merge_task->execute()) return true;
        else { state = State::NEED_FINISH; return true; }
    if (state == State::NEED_FINISH) { finish(); state = State::SUCCESS; }
    return false;
}

This code is simplified from MergePlainMergeTreeTask.cpp. Each call does one small part of the job. When the MergeTask says it is done, finish() commits the new part.

FutureMergedMutatedPart #

Both merges and mutations describe their result using FutureMergedMutatedPart. It is like a work order for the future part.

struct FutureMergedMutatedPart
{
    String name;
    MergeTreePartInfo part_info;
    DataPartsVector parts;
    MergeType merge_type = MergeType::Regular;
};

It knows which old parts will be merged and what the new part should be called.

MergeMutateSelectedEntry #

Before a task is created, the selected work is stored in a MergeMutateSelectedEntry. It protects the selected parts from being selected again by another background job.

struct MergeMutateSelectedEntry
{
    FutureMergedMutatedPartPtr future_part;
    CurrentlyMergingPartsTaggerPtr tagger;
    MutationCommandsConstPtr commands;
};

The tagger is a little lock that says: “These parts are busy right now.”

MutatePlainMergeTreeTask #

The mutation wrapper is similar, but it uses MutateTask instead of MergeTask. It also receives MutationCommands, which are the actual SET, DELETE, or ALTER operations.

Here is a simplified view:

class MutatePlainMergeTreeTask : public IExecutableTask
{
public:
    bool executeStep() override;
    void cancel() noexcept override;
private:
    MutateTaskPtr mutate_task;
};

When a mutation finishes, the new part replaces the old part. Then the mutation is marked as done for that part.

MergeTreeDataMergerMutator in Action #

The MergeTreeDataMergerMutator is the central service that creates these tasks.

In MergePlainMergeTreeTask::prepare(), you can see a call like this:

merge_task = storage.merger_mutator.mergePartsToTemporaryPart(
    future_part,
    metadata_snapshot,
    merge_list_entry,
    ...
);

The ellipsis hides many details, but the idea is simple: give the mutator a work order, and it returns a MergeTaskPtr.

In MutatePlainMergeTreeTask::prepare(), the same mutator creates a mutation task:

mutate_task = storage.merger_mutator.mutatePartToTemporaryPart(
    future_part,
    metadata_snapshot,
    merge_mutate_entry->commands,
    ...
);

Both paths follow the same pattern:

1. prepare the task,
2. execute it step by step,
3. commit the new part,
4. report the result.

A Note on Merges and Mutations in Replicated Tables #

This chapter described the non-replicated path. In a replicated MergeTree, the same tasks are used, but they are coordinated through ZooKeeper. The library example still works, but there is a second librarian on the other side of the building who needs to agree on what is being merged.

We will look at that coordination in the next chapter.

Summary #

In this chapter, you learned:

• Background jobs are implemented as coroutines with executeStep().
• MergeTreeDataMergerMutator builds MergeTask and MutateTask coroutines.
• Wrapper tasks like MergePlainMergeTreeTask and MutatePlainMergeTreeTask connect those coroutines to the background pool.
• MergeList tracks running merges and mutations in system.merges.
• Compaction selection is the policy side. IPartsCollector gathers active parts into ranges, MergeSelectorApplier uses SimpleMergeSelector or TTL selectors, and merge predicates check safety.
• Merges make reads faster. Mutations fix or modify data. Both happen in the background so your queries can keep running.

The MergeTree librarian is now working behind the scenes: combining thin booklets, applying corrections, and updating the library catalog. But what if there is more than one library in different cities? That is where replication comes in.

Continue to Chapter 6: Replication and ZooKeeper Coordination.

Happy querying!

Generated by AI Codebase Knowledge Builder