Elasticsearch, Logstash, and Kibana – The ELK stack

Conclusion

The ELK stack is not just useful for web or mail servers, where you can expect high hit counts, but also for large server clusters with distributed logs. Elasticsearch, Logstash, and Kibana are team players that collaborate excellently, and they are capable of integrating more components (e.g., services like Filebeat) into the team. The current program versions impress across the board and certainly give administrators a powerful toolbox.

Although the installation was a painless affair, the ELK stack does lack state-of-the-art systemd units in part, and even an init script in the case of Kibana. We also missed meaningful Logstash default configurations for the services on a typical Linux server. Although the documentation is very exhaustive, with many examples by other users on the web, it is a pity that system administrators first need to compile the information they need painstakingly. The commented examples on the FTP site [6] should be of help here.

The ELK stack is unbelievably flexible, but you can expect a lengthy learning curve. Many paths lead to a process chain. If the developers were to provide a basic set of configuration examples, they would help many admins and could help achieve initial results quickly, provide better orientation, and even allow admins to develop their own style.

Infos

  1. Elasticsearch: https://www.elastic.co/products/elasticsearch
  2. Logstash: https://www.elastic.co/products/logstash
  3. Kibana: https://www.elastic.co/products/kibana
  4. Filebeat: https://www.elastic.co/products/beats/filebeat
  5. Elastic subscriptions: https://www.elastic.co/subscriptions
  6. Listings in this article: ftp://ftp.linux-magazine.com/pub/listings/magazine/185
  7. Shield: https://www.elastic.co/products/shield
  8. Logstash reference: https://www.elastic.co/guide/en/logstash/current/index.html
  9. Logstash forwarder and Lumberjack protocol: https://github.com/elastic/logstash-forwarder
  10. Beats: https://www.elastic.co/products/beats
  11. Working with Logstash plugins: https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html
  12. Postfix patterns: https://github.com/whyscream/postfix-grok-patterns
  13. Dovecot patterns: https://github.com/augieschwer/grok-patterns
  14. Nginx patterns: https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413
  15. GitHub repository with Logstash patterns: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
  16. Grok debugger: http://grokdebug.herokuapp.com
  17. GrokConstructor: http://grokconstructor.appspot.com
  18. Curator: https://www.elastic.co/guide/en/elasticsearch/client/curator

The Author

Christian Rohmann is part of the DevOps team at NetCologne, an Internet service provider for the Cologne, Bonn, and Aachen area of Germany. Christian implemented a complete ELK stack there, which he uses above all to analyze and evaluate the Postfix, Dovecot, Apache, Nginx, and Open-Xchange logfiles.

« Previous 1 2 3 4

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
Print Issues
Digital Issues
 
SUBSCRIPTIONS
Print Subs
Digisubs
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • ELK Stack Workshop

    ELK Stack is a powerful monitoring system known for efficient log management and versatile visualization. This hands-on workshop will help you take your first steps with setting up your own ELK Stack monitoring solution.

    more »
  • Logstash

    When something goes wrong on a system, the logfile is the first place to look for troubleshooting clues. Logstash, a log server with built-in analysis tools, consolidates logs from many servers and even makes the data searchable.

    more »
  • Tutorials – Collectd

    The collectd tool harvests your system stats and stores them for plotting into colorful graphs.

    more »
  • Perl: Elasticsearch

    The Elasticsearch full-text search engine quickly finds expressions even in huge text collections. With a few tricks, you can even locate photos that have been shot in the vicinity of a reference image.

    more »
  • Perl – Elasticsearch

    Websites often offer readers links to articles about similar topics. Using Elasticsearch, the free search engine, is one way to find related documents instantly and automatically.

    more »
comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News

Tag Cloud

Administration Community Events Hardware Linux Linux Pro Magazine Mobile Programming Security Software Ubuntu Web Development Windows free software open source