OpenSearch® is a very robust solution when it comes to manage logs, thanks to its Lucene-based search engine and abilities to scale. Its native features makes log centralization, observability, monitoring and troubleshooting a breeze.

Understanding the Logging Stack

Even if it’s the main component, the OpenSearch® database itself is not enough to provide an efficient stack to manage logs. It’s often deployed with additional components to make it a full-featured logging stack:

• Logs Collector
  Retrieves logs from sources (applications, databases) and forwards them to the ingestion solution.
  Examples: Scalingo’s log drains, Filebeat
• Logs Ingestion
  Parses, enriches, transforms, filters, normalizes and aggregates log entries.
  Examples: Logstash, OpenSearch® Ingest Pipelines, OpenSearch® Data Prepper, Fluentd
• Storage
  Indexes and stores log entries.
  Examples: OpenSearch® database
• Visualization
  Allows to search, query and visualize the content of indexes.
  Examples: Kibana, OpenSearch® Dashboards
• Data Lifecyle Management
  Manages log entries, removes the oldest useless ones.
  Examples: Curator, OpenSearch® Index State Management

The following schema shows how these components are linked:

Planning your Deployment

If you don’t know where to start, we suggest to start with the following set of tools, which are known to work well on Scalingo and for which we have documentation: