↓Skip to main content
  1. Search Indices: From Protocol to Production/

Index Lifecycle: Visibility, Retention, and Aliases

Table of Contents

Index Lifecycle: Visibility, Retention, and Aliases #

An index does not stay in one state for its lifetime. For time-series data — logs, metrics, events — an index is created, filled with writes, rolled over to a new index, demoted through storage tiers as it ages, and eventually deleted. For mutable application data, the lifecycle is different: an index exists until its mapping becomes wrong, at which point it must be replaced via a coordinated swap.

The operational mechanisms for managing these transitions — ILM policies, aliases, rollover, force merge, and data streams — are the plumbing that keeps a search cluster healthy over time.

Index Lifecycle Management: Phases #

Index Lifecycle Management (ILM) is a policy engine. Each policy defines a set of phases, and indices transition through phases automatically when configured conditions are met.

Index Lifecycle Management phases: hot → warm → cold → frozen → delete

Hot — the active write-and-read phase. Indices in hot are on the highest-performance nodes (typically SSD-backed), receive all new documents, and serve the majority of search traffic. The hot phase triggers rollover: when the current index exceeds a size, age, or document count threshold, ILM creates a new index and advances the write alias to it.

Warm — the index is no longer receiving writes. ILM moves it to warm-tier nodes (typically lower-cost), optionally reduces its replica count, and may trigger a force merge to reduce segment count. Fewer segments → faster search; force merge is especially effective here because the index is read-only and will never produce new segments.

Cold — the index is accessed infrequently. It may be moved to even cheaper storage, with further reduced replicas. Search is still possible but latency is higher than warm.

Frozen — the index data is held in remote storage (S3-compatible object store). Only a minimal amount of data is cached locally. Searches load segments on demand and are significantly slower than warm. Frozen indices are appropriate for compliance retention where data must be searchable but is rarely queried.

Delete — the index is removed. Snapshots may be taken in a prior phase to preserve data in object storage before deletion.

PUT /_plugins/_ism/policies/orders-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "min_index_age": "7d",
            "min_primary_shard_size": "50gb",
            "min_doc_count": 100000000
          }
        }
      },
      "warm": {
        "min_index_age": "30d",
        "actions": {
          "replica_count": { "number_of_replicas": 1 },
          "force_merge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_index_age": "90d",
        "actions": {
          "replica_count": { "number_of_replicas": 0 }
        }
      },
      "delete": {
        "min_index_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

The min_index_age in each phase is measured from the index creation time (or rollover time if rollover_alias is set). ILM checks policy conditions on a configurable interval (indices.lifecycle.poll_interval, default 10 minutes).

Rollover: Controlled Index Growth #

Rollover is the mechanism that bounds index size. Rather than writing to a single index indefinitely, rollover creates a new index and redirects writes when conditions are met.

The rollover API checks conditions against the write alias target:

POST /orders/_rollover
{
  "conditions": {
    "max_age": "7d",
    "max_primary_shard_size": "50gb",
    "max_docs": 100000000
  }
}

When any condition is satisfied, OpenSearch:

  1. Creates orders-000002 (incrementing the numeric suffix).
  2. Applies the same index template settings (mappings, number_of_shards, ILM policy).
  3. Atomically moves the write alias (orders) from orders-000001 to orders-000002.
  4. Optionally keeps a read alias pointing to all backing indices.

After rollover, orders-000001 receives no new writes. It is available for reads via the read alias. ILM will move it through the subsequent phases according to its policy.

Why bound index size? Large indices have several failure modes: merges take longer and consume more disk I/O, recovery after a node failure takes longer (more segments to copy), shard rebalancing moves more data, and forced upgrades require longer maintenance windows. Keeping indices bounded — typically 10–50GB per primary shard — makes each of these operations tractable.

Aliases: Logical Names over Physical Indices #

An alias is a named pointer to one or more physical indices. All search and write operations can target aliases instead of index names, decoupling application logic from physical index structure.

Alias indirection: write alias → current index, read alias → all backing indices

Write alias — points to exactly one index at a time. All index/update/delete operations go to that one index. When rollover creates a new index, the write alias is atomically moved.

Read alias — can point to multiple indices simultaneously. A search against the read alias fans out across all backing indices — the scatter-gather read path is transparent to the caller.

POST /_aliases
{
  "actions": [
    { "add": { "index": "orders-000001", "alias": "orders",     "is_write_index": true } },
    { "add": { "index": "orders-000001", "alias": "orders-all" } }
  ]
}

After rollover:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "orders-000001", "alias": "orders" } },
    { "add":    { "index": "orders-000002", "alias": "orders",     "is_write_index": true } },
    { "add":    { "index": "orders-000002", "alias": "orders-all" } }
  ]
}

The _aliases action is atomic — no request sees orders pointing to zero indices or to two write indices simultaneously.

Data Streams: Opinionated Time-Series Abstraction #

A data stream is an abstraction built on top of rollover and aliases, designed for append-only time-series workloads. It provides the same rollover mechanics but enforces constraints that simplify operation:

  • Documents must have a @timestamp field.
  • All writes go to the current backing index (the stream’s write index).
  • Updates and deletes are supported but require the _id and operate via a separate mechanism.
  • The data stream name is the alias — there is no separate alias to manage.
PUT /_index_template/orders-data-stream-template
{
  "index_patterns": ["orders-stream"],
  "data_stream": {},
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" }
      }
    },
    "settings": {
      "index.lifecycle.name": "orders-policy"
    }
  }
}

PUT /_data_stream/orders-stream

Writes go to orders-stream with a standard POST /orders-stream/_doc/ request. Rollover happens automatically according to the attached ILM policy. The backing indices (orders-stream-000001, etc.) are managed by OpenSearch.

Data streams trade flexibility for operational simplicity. They are the right choice for log, metric, and event ingestion. They are the wrong choice for mutable data (frequent updates/deletes) or for data that does not have a meaningful @timestamp.

Segment Merges and Visibility #

A Lucene index (each OpenSearch shard) consists of multiple immutable segments. During active indexing, new segments are created continuously — one per refresh cycle. Lucene’s merge policy runs a background process that combines small segments into larger ones.

Merges affect query performance, not visibility. A merge does not make any document newly visible or remove any document from visibility. A merge takes N existing searchable segments and produces 1 new segment with the same documents. The new merged segment becomes visible atomically when the merge completes and a new index reader is opened.

Segment count affects query performance. Each segment must be independently searched during a query. An index with 500 small segments takes longer to query than one with 5 large segments with identical total document count. High segment counts arise after:

  • High-throughput indexing before merges catch up.
  • Frequent small refreshes.
  • Disabling automatic merges.

Force merge — manually trigger merges to a target segment count:

POST /orders-000001/_forcemerge?max_num_segments=1

max_num_segments: 1 reduces the entire shard to a single segment — optimal for read-only indices (warm, cold). Force merge on a write-active index is counterproductive: the merge competes with ongoing indexing for disk I/O, and new segments will be created immediately by continued writes.

Force merge is appropriate:

  • Before transitioning an index from hot to warm.
  • On a completed bulk import index.
  • Before a read-only index is exported as a snapshot.

Force merge is not appropriate:

  • On any index receiving active writes.
  • As a routine performance fix — address the root cause (refresh interval, indexing rate, merge policy settings) instead.

Shrink: Reducing Primary Shard Count #

An index’s primary shard count is set at creation and cannot be changed — splitting documents across N shards is baked into the routing hash. However, the Shrink API allows reducing the primary shard count post-creation under specific conditions.

Requirements:

  • All primary shards must be on the same node (use index.routing.allocation.require._name to force this).
  • The index must be read-only (index.blocks.write: true).
  • The target shard count must be a divisor of the source shard count.
PUT /orders-000001/_settings
{
  "index.routing.allocation.require._name": "warm-node-1",
  "index.blocks.write": true
}

POST /orders-000001/_shrink/orders-000001-shrunk
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1
  }
}

Shrink creates a new index with fewer shards by hard-linking the existing segment files. No data is copied — the new index’s segments reference the same underlying files as the original. This makes the operation fast.

When to shrink: during bulk indexing of a dataset, it is common to use many shards for write parallelism, then shrink to fewer shards once ingestion is complete and the index is read-only. This reduces the cluster’s shard count (a resource that contributes to cluster state overhead) without sacrificing the ingestion throughput that justified the original shard count.

Reindex: Copying Between Indices #

The _reindex API copies documents from a source index to a destination index. It is the mechanism for:

  • Correcting a mapping error (copy to a new index with correct mapping).
  • Changing the number of primary shards.
  • Upgrading index format across major versions.
  • Consolidating multiple indices.
POST /_reindex
{
  "source": {
    "index": "orders-v1",
    "query": { "range": { "created": { "gte": "2024-01-01" } } }
  },
  "dest": { "index": "orders-v2" },
  "conflicts": "proceed"
}

Reindex reads documents from the source and indexes them into the destination. The source is read live — documents written to the source after reindex starts may or may not be captured, depending on timing. For a clean migration:

  1. Pause writes to the source, or track the reindex start time.
  2. Run reindex.
  3. Index any documents written to the source after reindex started (query by modification timestamp).
  4. Swap the alias atomically.
  5. Resume writes via the new alias.

For large indices, reindex supports slices (parallel execution) and scroll batching:

POST /_reindex?slices=auto&wait_for_completion=false
{
  "source": { "index": "orders-v1" },
  "dest":   { "index": "orders-v2" }
}

slices=auto creates one slice per primary shard of the source. wait_for_completion=false returns a task ID immediately; poll progress with GET /_tasks/<task_id>.