Sensu — A powerful and scalable monitoring solution
Scaling
My company has now been using Sensu for more than a year on a dedicated server with state-of-the-art hardware, and we see no need for additional capacity. Although the infrastructure of the managed hosting platform is steadily increasing, the team has not once had such severely delayed checks as were experienced with Nagios using comparable hardware.
However, we still feed run-time metrics directly from the client to Graphite; if we were using Sensu as a metric distribution center, the traffic on the message bus would certainly increase significantly. We would probably then have to devote ourselves actively to the topic of scaling. Fortunately, Sensu is also well prepared for this.
The Sensu server as a central distribution point will, at some point, become the performance bottleneck with the increasing number of events. This problem could not be easier to solve: Simply connect other Sensu servers to RabbitMQ and Redis – and you're done. The Sensu servers automatically elect the master instance among themselves and distribute event processing – #monitoringlove! (See the "Who Monitors the Monitor?" box for more information.)
Who Monitors the Monitor?
From my own experience, I can say that it is a serious setback if, after a number of months full of #monitoringlove, you determine that errors have gone undetected for hours because Sensu has frozen. It is therefore important to monitor the function of Sensu itself. For example, a check, for which a single supervisor such as Monit is sufficient, should find out whether the central Sensu server process is running. Another check should investigate the Ready queue in RabbitMQ to identify processing backlogs.
Anyone using the Sensu API so intensively that it starts to moan under the load can counteract this problem with conventional load-balancing strategies, because Sensu uses a simple, stateless HTTP front end. Also, the RabbitMQ documentation describes strategies for scaling clusters that use the RabbitMQ message bus [10]. These approaches all aim to increase performance. You will have to consider each individual component separately if you want to make the entire Sensu system highly available [11].
Things become a little more complex if Sensu has to monitor large infrastructures that are distributed across multiple sites. Of course, a centralized approach is your first option (Figure 2). All Sensu clients perform their checks locally and deliver their results to the remote message bus via WAN.
However, intensive use of WAN connections also has a downside – in the form of much additional traffic. An unstable network connection can easily lead to a flood of host-down alarms.
An architecture with distributed instances of RabbitMQ, as shown in Figure 3, can help ensure that no events are lost on the (WAN) link. The Federation plugin [12] and the Shovel plugin [13] for RabbitMQ can help you implement this setup.
This approach favors the characteristics of availability and partition tolerance over consistency; however, it also suffers from a weakness that can cause keepalive alerts in case of connection problems. You can handle this problem by setting up check dependencies.
Reliance on a stable WAN connection is lifted if you run not just the message bus but the complete Sensu system locally at each site. A common view of the overall state of our infrastructure is maintained thanks to the fact that the Sensu dashboard can represent the states of multiple Sensu installations.
Figure 4 graphically depicts this aggregation approach. This redundancy may mean slightly higher installation overhead, but users can leave this to a configuration management system such as Chef.
Conclusions
Sensu is highly suited for its role as the central point of a comprehensive monitoring system. The Sensu monitoring tool is easy to operate and can easily reuse existing checks thanks to the Nagios plugin interface. Sensu also lets you integrate a steadily growing number of external services, and it is well equipped for the future through its scalability. #monitoringsucks is finally a thing of the past.
Infos
- Sensu: https://github.com/sensu
- Sensu admin: https://github.com/sensu/sensu-admin
- Chef cookbook: https://github.com/sensu/sensu-chef
- Puppet module: https://github.com/sensu/sensu-puppet
- Sensu shell helper: https://github.com/solarkennedy/sensu-shell-helper
- Sensu plugins: https://github.com/sensu/sensu-community-plugins
- Sensu CLI: https://github.com/agent462/sensu-cli
- Hubot: https://github.com/sensu/sensu-hubot
- Lita: https://www.lita.io
- RabbitMQ, Cluster documentation: https://www.RabbitMQ.com/clustering.html
- "High availability monitoring with Sensu" by Jean-Francois Theroux: http://failshell.io/sensu/high-availability-sensu/
- Federation plugin: https://www.RabbitMQ.com/federation.html
- Shovel plugin: https://www.RabbitMQ.com/shovel.html
« Previous 1 2 3
Buy this article as PDF
(incl. VAT)