OpenSearch
An open-source search and analytics engine (forked from Elasticsearch 7.10) used for full-text search, log analytics, observability, and security analytics workloads at scale.
What is OpenSearch?
OpenSearch is an open-source, distributed search and analytics engine maintained by AWS and a community of contributors, forked from Elasticsearch 7.10 in 2021 following Elastic's licence change. It uses an inverted index architecture for full-text search and supports structured queries, aggregations, vector search (k-NN), and real-time analytics — making it suitable for search, log analytics, SIEM, and observability use cases.
OpenSearch clusters scale horizontally by adding nodes, distributing shards across the cluster. Index design — shard count, mapping, field data types, and ingest pipeline configuration — determines both query latency and storage efficiency. Over-sharded indexes (too many small shards) degrade cluster health; under-sharded indexes (too few large shards) limit parallelism and query throughput.
AiRAT migrated a major retail client from a 15-node Elasticsearch cluster to OpenSearch, achieving sub-40ms p99 query latency on a 2TB product catalog with 8M daily queries. The migration involved index re-mapping, ingest pipeline refactoring, and a blue-green cutover strategy that delivered zero downtime across a four-hour migration window. For SOC deployments, AiRAT uses OpenSearch as the SIEM backend — ingesting, indexing, and querying security telemetry at petabyte scale.
Examples in production
Product search at scale
2TB product catalogue indexed across 12 OpenSearch shards, serving 8M daily queries with sub-40ms p99 latency using custom relevance scoring, synonym expansion, and query-time boosting for personalised results.
Security analytics backend
OpenSearch ingests 500GB/day of normalised security telemetry from 40 log sources, enabling analysts to run correlation queries across 90 days of data in under 5 seconds — replacing a legacy SIEM that required 40-second query times.
Questions about OpenSearch
What is the difference between OpenSearch and Elasticsearch?
OpenSearch is a community-maintained open-source fork of Elasticsearch 7.10, released under Apache 2.0 licence. Elasticsearch continues development under the Elastic Source Licence (SSPL/ELv2), which restricts offering it as a managed service. For self-hosted or AWS-managed deployments, OpenSearch is functionally equivalent and actively developed. For Elastic Cloud managed deployments, Elasticsearch is the natural choice. AiRAT works with both.
How many shards should an OpenSearch index have?
A common starting point: aim for shard sizes of 20–50GB for search workloads and 10–30GB for log analytics. Over-sharding increases cluster overhead (more shard metadata, more coordination); under-sharding limits query parallelism. For time-series data (logs, events), use daily or weekly indices with ILM (Index Lifecycle Management) to roll, shrink, and delete old indices automatically.
Running a system that uses OpenSearch?
We build production-grade data engineering systems for regulated enterprises.