Elasticsearch Index Lifecycle Management — Automate Hot-Warm-Cold Architectures

The Index Sprawl Problem

A production Elasticsearch cluster ingesting log or metric data will create hundreds of indices over its lifetime. Without automation, operations teams end up manually curating storage: deleting old indices when disks fill up, moving cold data to cheaper nodes by hand, force-merging forgotten segments that consume file handles. This is not a workflow — it is a liability.

Index Lifecycle Management (ILM) is Elasticsearch's built-in policy engine for automating exactly this. Define a policy once — rollover at 50 GB, warm after 7 days, cold after 30 days, delete after 90 — and every index governed by that policy moves through its lifecycle without human intervention. When paired with a tiered node architecture, ILM can cut Elasticsearch storage costs by 60–80% while maintaining sub-second query latency for recent data.

The Five ILM Phases Explained

ILM organizes an index's life into up to five sequential phases. Not every policy uses all five — most production setups use hot, warm, and delete at minimum, adding cold or frozen for cost-sensitive long-term retention.

Configuring Node Tiers and Data Roles

ILM's tier migration actions work by routing shards to nodes with specific data tier roles. Each node in your cluster declares which tiers it belongs to in elasticsearch.yml. Nodes can hold multiple roles, but production clusters should keep hot and warm nodes separate for hardware and performance isolation.

# elasticsearch.yml — hot-tier node
# Dedicated hot nodes: NVMe SSDs, high IOPS, all memory for indexing
node.roles: [ data_hot, ingest ]
node.name: hot-01
path.data: /mnt/nvme/elasticsearch

---

# elasticsearch.yml — warm-tier node
# Dedicated warm nodes: SATA SSDs, more storage per dollar
node.roles: [ data_warm ]
node.name: warm-01
path.data: /mnt/sata/elasticsearch

---

# elasticsearch.yml — cold-tier node
# Dedicated cold nodes: spinning disks or high-density SSDs
node.roles: [ data_cold ]
node.name: cold-01
path.data: /mnt/hdd/elasticsearch

---

# elasticsearch.yml — frozen-tier node
# Minimal local cache — data lives in object storage (S3/GCS/Azure Blob)
node.roles: [ data_frozen ]
node.name: frozen-01
# Frozen nodes need a snapshot cache directory
path.data: /mnt/cache/elasticsearch

---

# elasticsearch.yml — content/master nodes (not part of data tiers)
node.roles: [ master, data_content ]
node.name: master-01

Writing Your First ILM Policy

ILM policies are defined as JSON documents and stored in the Elasticsearch cluster state. They can be created via the REST API, Kibana's Stack Management > Index Lifecycle Policies UI, or infrastructure-as-code tools like the elasticstack Terraform provider. The REST API is the most explicit and reproducible approach.

# Create a production ILM policy for application logs
# 7 days hot → 30 days warm → 60 days cold → delete at 90 days

PUT _ilm/policy/logs-production
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d",
            "max_docs": 100000000
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "migrate": {},
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "migrate": {},
          "searchable_snapshot": {
            "snapshot_repository": "s3-logs-archive",
            "force_merge_index": true
          },
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Rollover: The Core of Hot-Phase Management

Rollover is the action that creates a new write index and marks the old one as read-only. It is triggered when any one of the configured conditions is met — size, document count, or age. Setting multiple conditions is additive (OR logic), so the policy rolls over whichever threshold is crossed first.

ILM rollover requires an index alias or data stream with a write target. Without one, ILM has no way to know which index to write to after rollover. The conventional naming pattern is logs-app-000001, logs-app-000002, etc., with the alias logs-app pointing to the current write index.

# Bootstrap the first index with proper ILM settings and alias
# Run this once before your first document ingestion

PUT logs-app-000001
{
  "aliases": {
    "logs-app": {
      "is_write_index": true
    }
  },
  "settings": {
    "index": {
      "lifecycle": {
        "name": "logs-production",
        "rollover_alias": "logs-app"
      },
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "routing.allocation.include._tier_preference": "data_hot"
    }
  }
}

# Verify ILM is tracking the index
GET logs-app-000001/_ilm/explain

# Example output from _ilm/explain — healthy ILM tracking
{
  "indices": {
    "logs-app-000001": {
      "index": "logs-app-000001",
      "managed": true,
      "policy": "logs-production",
      "lifecycle_date_millis": 1745000000000,
      "phase": "hot",
      "phase_time_millis": 1745000000000,
      "action": "rollover",
      "action_time_millis": 1745000100000,
      "step": "check-rollover-ready",
      "step_time_millis": 1745000200000,
      "phase_execution": {
        "policy": "logs-production",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "1d"
            }
          }
        }
      }
    }
  }
}

Attaching ILM to Index Templates

In production, you should not configure ILM settings on individual indices by hand. Instead, use composable index templates to inject ILM policy, shard count, and tier routing automatically for any index matching a pattern. This ensures every new index — whether created by rollover, Beats, Logstash, or a data stream — inherits the correct lifecycle settings without manual intervention.

# Step 1: Create a component template for ILM settings
# Component templates are reusable building blocks

PUT _component_template/logs-ilm-settings
{
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-production",
      "index.lifecycle.rollover_alias": "logs-app",
      "index.number_of_shards": 2,
      "index.number_of_replicas": 1,
      "index.routing.allocation.include._tier_preference": "data_hot"
    }
  }
}

# Step 2: Create a component template for mappings
PUT _component_template/logs-mappings
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp":    { "type": "date" },
        "log.level":     { "type": "keyword" },
        "service.name":  { "type": "keyword" },
        "trace.id":      { "type": "keyword" },
        "message":       { "type": "text", "norms": false },
        "http.status":   { "type": "short" },
        "http.method":   { "type": "keyword" },
        "url.path":      { "type": "keyword" },
        "duration_ms":   { "type": "float" }
      }
    }
  }
}

# Step 3: Create the composable index template combining both components
PUT _index_template/logs-app-template
{
  "index_patterns": ["logs-app-*"],
  "composed_of": ["logs-ilm-settings", "logs-mappings"],
  "priority": 200,
  "template": {
    "settings": {
      "index.codec": "best_compression"
    }
  },
  "_meta": {
    "description": "Production logs — ILM managed, hot-warm-cold",
    "team": "platform-engineering"
  }
}

Data Streams: The Modern Approach

For append-only time-series data, data streams are the recommended abstraction over manual alias management. A data stream is a named abstraction that routes writes to the current backing index and automatically manages rollover via ILM — without requiring you to create the first index, manage aliases, or track write targets manually.

Data streams also enforce the @timestamp field requirement and expose a clean API for querying across all backing indices. Kibana, Beats, Elastic Agent, and the OpenTelemetry Collector all write to data streams natively.

# Create a data stream index template (no alias bootstrapping required)
# ILM automatically manages backing indices

PUT _index_template/metrics-services-template
{
  "index_patterns": ["metrics-services-*"],
  "data_stream": {},
  "composed_of": ["metrics-ilm-settings", "metrics-mappings"],
  "priority": 200
}

# Create the data stream — ILM creates the first backing index automatically
PUT _data_stream/metrics-services-production

# Write to the data stream (always routed to the current write index)
POST metrics-services-production/_doc
{
  "@timestamp": "2026-04-19T12:00:00Z",
  "service.name": "payment-api",
  "http.status": 200,
  "duration_ms": 45.3
}

# Inspect backing indices and ILM phase for each
GET _data_stream/metrics-services-production

# Manually trigger rollover (useful for testing ILM transitions)
POST metrics-services-production/_rollover

Monitoring ILM Execution and Troubleshooting

ILM checks policies on a configurable interval (default: 10 minutes). Steps within a phase execute asynchronously — a force-merge on a large shard may take hours. Understanding how to inspect ILM state is critical for diagnosing stuck transitions, failed actions, and unexpected rollover behavior.

# Check ILM status for all managed indices
GET */_ilm/explain?human=true

# Check a specific index with human-readable timestamps
GET logs-app-000005/_ilm/explain?human=true

# ILM step failure — look for "failed_step" and "step_info"
# Example output when an action fails:
# {
#   "phase": "warm",
#   "action": "forcemerge",
#   "step": "forcemerge",
#   "failed_step": "forcemerge",
#   "step_info": {
#     "type": "exception",
#     "reason": "primary shard is not active"
#   }
# }

# Retry a failed step after fixing the underlying issue
POST logs-app-000005/_ilm/retry

# Change the ILM poll interval (default 10m — reduce for testing)
PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "1m"
  }
}

# Move an index to a specific phase/step manually (for testing or recovery)
POST _ilm/move/logs-app-000005
{
  "current_step": {
    "phase": "warm",
    "action": "complete",
    "name": "complete"
  },
  "next_step": {
    "phase": "cold",
    "action": "migrate",
    "name": "migrate"
  }
}

Managing ILM Policies Programmatically

For infrastructure-as-code workflows, the official Elasticsearch Python client exposes the full ILM API. The pattern below is suitable for CI/CD pipelines that apply ILM policies as part of cluster bootstrapping or upgrades.

from elasticsearch import Elasticsearch
from elasticsearch.exceptions import NotFoundError
import json

es = Elasticsearch(
    hosts=["https://elasticsearch:9200"],
    api_key=("key-id", "key-secret"),  # prefer API key auth in CI
)


def upsert_ilm_policy(name: str, policy: dict) -> None:
    """Create or update an ILM policy — idempotent."""
    try:
        existing = es.ilm.get_lifecycle(name=name)
        current_policy = existing[name]["policy"]
        if current_policy == policy:
            print(f"ILM policy '{name}' is up to date — no changes.")
            return
    except NotFoundError:
        pass

    es.ilm.put_lifecycle(name=name, policy=policy)
    print(f"ILM policy '{name}' applied.")


LOGS_POLICY = {
    "phases": {
        "hot": {
            "min_age": "0ms",
            "actions": {
                "rollover": {
                    "max_primary_shard_size": "50gb",
                    "max_age": "1d",
                },
                "set_priority": {"priority": 100},
            },
        },
        "warm": {
            "min_age": "7d",
            "actions": {
                "migrate": {},
                "shrink": {"number_of_shards": 1},
                "forcemerge": {"max_num_segments": 1},
                "set_priority": {"priority": 50},
            },
        },
        "cold": {
            "min_age": "30d",
            "actions": {
                "migrate": {},
                "searchable_snapshot": {
                    "snapshot_repository": "s3-logs-archive",
                    "force_merge_index": True,
                },
                "set_priority": {"priority": 0},
            },
        },
        "delete": {
            "min_age": "90d",
            "actions": {
                "delete": {"delete_searchable_snapshot": True},
            },
        },
    }
}

upsert_ilm_policy("logs-production", LOGS_POLICY)


def list_indices_by_phase(policy_name: str) -> dict[str, list[str]]:
    """Group managed indices by their current ILM phase."""
    response = es.ilm.explain_lifecycle(index="*")
    by_phase: dict[str, list[str]] = {}
    for index_name, info in response["indices"].items():
        if not info.get("managed"):
            continue
        if info.get("policy") != policy_name:
            continue
        phase = info.get("phase", "unknown")
        by_phase.setdefault(phase, []).append(index_name)
    return by_phase


phases = list_indices_by_phase("logs-production")
for phase, indices in sorted(phases.items()):
    print(f"{phase:10s}: {len(indices)} indices")
    for idx in sorted(indices)[:5]:  # show first 5 per phase
        print(f"           {idx}")

Searchable Snapshots: Cost-Efficient Cold and Frozen Storage

Searchable snapshots store index data in an object-storage repository (Amazon S3, Google Cloud Storage, Azure Blob Storage, or self-hosted S3-compatible stores like MinIO) while making it queryable without full restore. In the cold tier, Elasticsearch caches the most-accessed segment files locally. In the frozen tier, only a minimal metadata cache is maintained — each query fetches the required segments on demand.

Enabling searchable snapshots for ILM requires a registered snapshot repository. The repository is created once per cluster and referenced by name in the ILM policy.

# Register an S3 snapshot repository
# Requires the repository-s3 plugin and AWS credentials in the keystore

PUT _snapshot/s3-logs-archive
{
  "type": "s3",
  "settings": {
    "bucket": "my-elasticsearch-snapshots",
    "region": "eu-west-1",
    "base_path": "logs-archive",
    "compress": true,
    "server_side_encryption": true,
    "storage_class": "standard_ia"  // S3 Infrequent Access — cheaper for cold data
  }
}

# Verify the repository is reachable
POST _snapshot/s3-logs-archive/_verify

# ILM cold phase with searchable snapshots enabled
# (reference the same repository name in your policy)
# "cold": {
#   "min_age": "30d",
#   "actions": {
#     "searchable_snapshot": {
#       "snapshot_repository": "s3-logs-archive"
#     }
#   }
# }

# Check a mounted searchable snapshot index
GET .ds-logs-app-000003/_stats?human=true

# Searchable snapshot indices have a special _tier value
GET .ds-logs-app-000003/_settings?filter_path=**.routing.allocation

Production Best Practices