OpenSearch Operations, Replication, Sharding, and HA

OpenSearch is a distributed search engine. It stores data in indexes, splits indexes into primary shards, copies those shards to replica shards, and coordinates cluster state through elected cluster-manager nodes. High availability comes from shard allocation, node placement, cluster-manager quorum, snapshots, and tested operational procedures. It does not come from replicas alone.

For concrete cluster health, shard, allocation, and index-template examples, see Database and Search Examples.

Command Examples

curl -sS https://<opensearch>:9200/_cluster/health?pretty
curl -sS https://<opensearch>:9200/_cat/nodes?v
curl -sS https://<opensearch>:9200/_cat/shards?v
curl -sS https://<opensearch>:9200/_cluster/allocation/explain?pretty -H 'Content-Type: application/json' -d '{}'
curl -sS https://<opensearch>:9200/_cat/indices?v
curl -sS https://<opensearch>:9200/_plugins/_replication/<follower-index>/_status?pretty

Example output and meaning:

These checks separate cluster health, node roles, shard placement, allocation blockers, index state, and cross-cluster replication state. Pair them with JVM heap, disk watermarks, CPU, network, thread pools, and ingest error rates.

Core Model

Cluster health colors are shard-allocation signals:

• Green: all primary and replica shards are allocated.
• Yellow: all primary shards are allocated, but at least one replica is unassigned.
• Red: at least one primary shard is unassigned, so some data is unavailable.

Yellow is not harmless. It means the cluster can serve the data whose primaries are allocated, but it has reduced redundancy and may not survive another node or disk failure for affected indexes.

Sharding

A shard is a Lucene index. OpenSearch routes each document to one primary shard, usually by hashing the document ID or a custom routing value. The primary shard handles the write, then replication makes the write available on replica shard copies.

Important sharding settings:

Shard-count design is a capacity decision:

• Too few primary shards can cap indexing and search fan-out on a large index.
• Too many shards increase heap overhead, file handles, cluster-state size, recovery work, and search coordination cost.
• A replica is not a partition. More replicas can improve read capacity and redundancy, but they also increase disk use and replication work.
• Use rollover and index lifecycle patterns for time-series data instead of creating one ever-growing index.
• Prefer aliases for application access so reindexing, rollover, and shard-count changes do not require application changes.

Common sharding patterns:

Operational sharding rules:

• Choose primary shard count from expected index size, write rate, query patterns, and recovery time, not from a fixed rule copied across systems.
• Keep primary and replica copies on different nodes and preferably different zones/racks.
• Use allocation awareness for failure domains. Without it, replicas may concentrate where they do not protect you.
• Use forced awareness when you would rather leave replicas unassigned than place both copies in the same failure domain.
• Use _cluster/allocation/explain before changing allocation settings during an incident.
• Watch shard skew: largest shards, hottest shards, slowest shards, and shards stuck initializing or relocating.

In-Cluster Replication

OpenSearch replication inside one cluster is shard-copy replication. Each primary shard has zero or more replica shard copies depending on index.number_of_replicas.

Write path:

1. A client sends a write to any node.
2. The coordinating node routes the request to the primary shard for that document.
3. The primary validates and applies the write.
4. Replica copies receive and apply the operation.
5. The client response reports how many shard copies succeeded or failed.

wait_for_active_shards can require a minimum number of active shard copies before an operation starts. It reduces the chance of writing while the cluster is under-replicated, but it is not the same as a global durability guarantee. Replication can still fail after the operation begins, and the response must be checked.

Replication modes and related features:

Replica count guidance:

• 0 replicas is acceptable only for disposable data or development.
• 1 replica is the common minimum for production indexes on multi-node clusters.
• More replicas help read fan-out and failure tolerance, but they increase disk, network, recovery, and indexing overhead.
• Replicas do not protect against deletes, bad mappings, bad ingest pipelines, ransomware, or application bugs. Use snapshots.

Cross-Cluster Replication

Cross-cluster replication replicates indexes from a leader cluster to one or more follower indexes in another cluster. It is active-passive: writes happen on the leader index, and follower indexes are read-only copies that pull changes from the leader.

Use cases:

• regional disaster recovery,
• local reads near users,
• central reporting from smaller clusters,
• migration staging,
• keeping a warm search cluster ready for controlled failover.

Important properties:

• Both clusters need the replication plugin.
• The follower side needs remote-cluster connectivity to the leader.
• If node roles are explicitly configured, follower nodes that connect remotely need the remote_cluster_client role.
• Security plugin posture and replication permissions must be designed on both clusters.
• Replication is per index or index pattern through auto-follow rules, not automatic whole-cluster magic.
• Follower indexes are read-only while following.

Cross-cluster replication operations:

Failover with cross-cluster replication is not automatic application magic. A controlled regional failover needs:

1. Confirm follower indexes are current enough for the intended RPO.
2. Stop writes to the old leader or declare it lost.
3. Stop replication or promote the follower-side data according to the runbook.
4. Move application writes and reads to the new cluster endpoint or alias.
5. Prevent the old leader from accepting divergent writes when it returns.
6. Rebuild reverse replication or re-seed the old cluster before failback.

HA Failover

OpenSearch has several different failover planes. Keep them separate.

Cluster-manager HA:

• Use three dedicated cluster-manager-eligible nodes for most production clusters.
• Spread them across failure domains.
• Avoid two cluster-manager nodes as a production quorum design. A two-node voting set cannot safely tolerate one side of a partition without risking split brain.
• Keep data-heavy and ingest-heavy work off dedicated cluster-manager nodes.

Data-node HA:

• Every production index that must survive node loss needs replicas and enough nodes to place them away from primaries.
• Disk watermarks, allocation filters, awareness rules, and node roles can prevent replicas from allocating even when raw disk exists.
• A red index means at least one primary shard is unavailable. A yellow index means primaries exist but redundancy is missing.
• Restoring missing primaries generally means recovering the original node, allocating a stale primary only when you accept data loss, or restoring from snapshot.

HA runbook for a failed data node:

1. Check _cluster/health, _cat/shards, _cat/nodes, and allocation explanation.
2. Determine whether the node is truly gone, slow, partitioned, disk-full, or blocked by JVM pressure.
3. If primaries are active and only replicas are unassigned, keep serving but reduce risk and restore redundancy.
4. If primaries are unassigned, decide whether the missing node can return quickly or whether snapshot restore is required.
5. Check allocation blockers: disk watermarks, awareness, forced awareness, include/exclude filters, index blocks, and node roles.
6. Avoid forcing stale primary allocation unless the data-loss decision is explicit and documented.
7. After recovery, verify shard balance, search results, ingest success, snapshots, and alerts.

Snapshots and Recovery

Snapshots are backups stored outside the cluster. Replicas are for availability; snapshots are for recovery from corruption, deletion, bad rollout, cluster loss, or accidental writes.

Snapshot rules:

• Store snapshots outside the cluster failure domain.
• Automate snapshot creation and retention.
• Test restore into a separate cluster.
• Include cluster state when needed, but understand what settings/templates you are restoring.
• Do not assume a green cluster means backups work.
• During disaster recovery, restore critical indexes first instead of waiting for every historical index if RTO matters.

Recovery choices:

Study Cards

References

• OpenSearch cluster health API
• OpenSearch index settings
• OpenSearch shard allocation filtering
• OpenSearch cluster allocation explain API
• OpenSearch cross-cluster replication
• OpenSearch cross-cluster replication getting started
• OpenSearch snapshots
• OpenSearch voting and quorum

curl -s localhost:9200/_cluster/health?pretty

{
  "status" : "yellow",
  "active_shards_percent_as_number" : 94.2,
  "number_of_pending_tasks" : 37
}

curl -s localhost:9200/_cat/shards?v

index    shard prirep state      docs  store ip          node
logs-01  2     r      UNASSIGNED
logs-01  2     p      STARTED    12m   42gb  10.0.1.12  os-node-1

curl -s localhost:9200/_cat/allocation?v

shards disk.indices disk.used disk.avail disk.percent host       node
318    1.9tb        2.7tb     110gb      96           10.0.1.12  os-node-1