System monitoring for a new generation with Prometheus
Push Gateway
All services cannot be meaningfully run with exporters. This is particularly problematic for services that generate data on the fly and need to offload the data immediately. One classic example is Cron, which handles a specific task and generates output while doing so, which Prometheus then needs to process. Prometheus has the push gateway to handle this kind of situation.
The gateway runs as a separate service with a RESTful interface and waits for the required values to arrive. If you want to deliver the Cron job output to the push gateway, all you need to do is append a pipe and a Curl call to the Cron job. The REST interface on the push gateway accesses the target host for Curl.
Of course, these gateways will scale horizontally; in other words you can run multiple gateways in parallel in the same installation. The Prometheus servers pick up the stored data from the gateways in exactly the same way they do with normal exporters. But, there is one slight drawback: The data output by the Cron job needs to exist in a format that the push gateway actually understands. In other words, you may need to tune the Cron output.
Alert Manager
If something is going wrong in the cluster, administrators would naturally expect the monitoring system to tell them. Prometheus has two possible approaches. Prometheus itself offers a basic alerting function: if the value of a metric is outside of defined boundaries over a defined period of time, Prometheus sends a message to a previously defined address.
However, the developers expressly point out that this function is only basic alerting, and the number of tweaks required for modifying the alerting style to meet your local needs limits flexibility.
But Prometheus offers a second approach: instead of handing alerting itself, Prometheus can forward alerts directly to the Alert Manager. The Alert Manager is a separate component in the Prometheus universe (Figure 4), and it is probably one of the biggest construction sites in Prometheus right now.
When this issue went to press, the Alert Manager GitHub page [8] featured a statement to the effect that the version in the master branch is a totally new development that is no longer compatible with the current stable version. If you want to take a look at, or test, Prometheus, you will still want to go for this version: the legacy version will disappear in the foreseeable future.
As an interface to Prometheus, the Alert Manager has its own API. When the Alert Manager receives an alert from Prometheus, it can process that alert in a variety of ways: the manager can mail a warning or forward the alert to the Pager Duty API.
The pager, in turn, supports various modules. Out of the box are Pager Duty modules for Slack, Hipchat, and various other chat protocols. But you could hitch up Pager Duty to a text service with a matching SMS module.
PromDash
PromDash (Figure 5) is a powerful tool for rendering the metrics stored in Prometheus in a meaningful way. The tool's developers describe PromDash as a dashboard builder, thus setting it apart from standard dashboards.
PromDash does not try to give users prebuilt templates for charts but instead gives administrators a tool for creating these templates themselves. PromDash can thus rightly be described as a graphical helper that facilitates the use of the PromQL language.
After setting up the solution, you need to add the servers that will act as data sources in PromDash, then sit back and enjoy the ride. One of PromDash's features is the ability to annotate graphs or integrate them directly into other pages. For example, you could easily implement a status dashboard for the entire platform that would give you a centralized overview of all the systems you need to monitor.
The solution for configuring PromDash itself is also very smart: The PromDash module stores the configured dashboards in a MySQL database in the background. This design means the configuration and program are separate, so you can run any number of PromDash instances on the same host without you needing to manually synchronize the configurations.
« Previous 1 2 3 4 Next »
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.