Sensu — A powerful and scalable monitoring solution

Scaling

My company has now been using Sensu for more than a year on a dedicated server with state-of-the-art hardware, and we see no need for additional capacity. Although the infrastructure of the managed hosting platform is steadily increasing, the team has not once had such severely delayed checks as were experienced with Nagios using comparable hardware.

However, we still feed run-time metrics directly from the client to Graphite; if we were using Sensu as a metric distribution center, the traffic on the message bus would certainly increase significantly. We would probably then have to devote ourselves actively to the topic of scaling. Fortunately, Sensu is also well prepared for this.

The Sensu server as a central distribution point will, at some point, become the performance bottleneck with the increasing number of events. This problem could not be easier to solve: Simply connect other Sensu servers to RabbitMQ and Redis – and you're done. The Sensu servers automatically elect the master instance among themselves and distribute event processing – #monitoringlove! (See the "Who Monitors the Monitor?" box for more information.)

Who Monitors the Monitor?

From my own experience, I can say that it is a serious setback if, after a number of months full of #monitoringlove, you determine that errors have gone undetected for hours because Sensu has frozen. It is therefore important to monitor the function of Sensu itself. For example, a check, for which a single supervisor such as Monit is sufficient, should find out whether the central Sensu server process is running. Another check should investigate the Ready queue in RabbitMQ to identify processing backlogs.

Anyone using the Sensu API so intensively that it starts to moan under the load can counteract this problem with conventional load-balancing strategies, because Sensu uses a simple, stateless HTTP front end. Also, the RabbitMQ documentation describes strategies for scaling clusters that use the RabbitMQ message bus [10]. These approaches all aim to increase performance. You will have to consider each individual component separately if you want to make the entire Sensu system highly available [11].

Things become a little more complex if Sensu has to monitor large infrastructures that are distributed across multiple sites. Of course, a centralized approach is your first option (Figure 2). All Sensu clients perform their checks locally and deliver their results to the remote message bus via WAN.

Figure 2: The structure of a solution for distributed monitoring with a centralized Sensu instance.

However, intensive use of WAN connections also has a downside – in the form of much additional traffic. An unstable network connection can easily lead to a flood of host-down alarms.

An architecture with distributed instances of RabbitMQ, as shown in Figure 3, can help ensure that no events are lost on the (WAN) link. The Federation plugin [12] and the Shovel plugin [13] for RabbitMQ can help you implement this setup.

Figure 3: An approach for a distributed Sensu architecture.

This approach favors the characteristics of availability and partition tolerance over consistency; however, it also suffers from a weakness that can cause keepalive alerts in case of connection problems. You can handle this problem by setting up check dependencies.

Reliance on a stable WAN connection is lifted if you run not just the message bus but the complete Sensu system locally at each site. A common view of the overall state of our infrastructure is maintained thanks to the fact that the Sensu dashboard can represent the states of multiple Sensu installations.

Figure 4 graphically depicts this aggregation approach. This redundancy may mean slightly higher installation overhead, but users can leave this to a configuration management system such as Chef.

Figure 4: Aggregation approach for Sensu in geographically distributed environments.

Conclusions

Sensu is highly suited for its role as the central point of a comprehensive monitoring system. The Sensu monitoring tool is easy to operate and can easily reuse existing checks thanks to the Nagios plugin interface. Sensu also lets you integrate a steadily growing number of external services, and it is well equipped for the future through its scalability. #monitoringsucks is finally a thing of the past.

The Author

Jochen Lillich is the Managing Director of freistil IT, which operates a hosting platform for business-critical Drupal and WordPress sites on freistilbox.com. He has focused on professional use of open source software for more than 20 years. Jochen lives with his family outside the Irish capital, Dublin, and manages his internationally distributed team from there.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Prometheus

    Legacy monitoring solutions are fine for small-to-medium-sized networks, but complex environments benefit from a different approach. Prometheus is an interesting alternative to classic tools like Nagios.

  • IoT with RabbitMQ

    Connect multiple protocols and servers together on your IoT projects.

  • ELK Stack Workshop

    ELK Stack is a powerful monitoring system known for efficient log management and versatile visualization. This hands-on workshop will help you take your first steps with setting up your own ELK Stack monitoring solution.

  • Programming Snapshot – Alexa

    Asking Alexa only for built-in functions like the weather report gets old quickly, and add-on skills from the skills store only go so far. With a few lines of code, Mike teaches this digital pet some new tricks.

  • Perl: Network Monitoring

    To discover possibly undesirable arrivals and departures on their networks, a Perl daemon periodically stores the data from Nmap scans and passes them on to Nagios via a built-in web interface.

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News