Push Gateway

All services cannot be meaningfully run with exporters. This is particularly problematic for services that generate data on the fly and need to offload the data immediately. One classic example is Cron, which handles a specific task and generates output while doing so, which Prometheus then needs to process. Prometheus has the push gateway to handle this kind of situation.

The gateway runs as a separate service with a RESTful interface and waits for the required values to arrive. If you want to deliver the Cron job output to the push gateway, all you need to do is append a pipe and a Curl call to the Cron job. The REST interface on the push gateway accesses the target host for Curl.

Of course, these gateways will scale horizontally; in other words you can run multiple gateways in parallel in the same installation. The Prometheus servers pick up the stored data from the gateways in exactly the same way they do with normal exporters. But, there is one slight drawback: The data output by the Cron job needs to exist in a format that the push gateway actually understands. In other words, you may need to tune the Cron output.

Alert Manager

If something is going wrong in the cluster, administrators would naturally expect the monitoring system to tell them. Prometheus has two possible approaches. Prometheus itself offers a basic alerting function: if the value of a metric is outside of defined boundaries over a defined period of time, Prometheus sends a message to a previously defined address.

However, the developers expressly point out that this function is only basic alerting, and the number of tweaks required for modifying the alerting style to meet your local needs limits flexibility.

But Prometheus offers a second approach: instead of handing alerting itself, Prometheus can forward alerts directly to the Alert Manager. The Alert Manager is a separate component in the Prometheus universe (Figure 4), and it is probably one of the biggest construction sites in Prometheus right now.

Figure 4: With the right choice of metrics and matching parameters, you can reach a "good" or "bad" decision for any situation.

When this issue went to press, the Alert Manager GitHub page [8] featured a statement to the effect that the version in the master branch is a totally new development that is no longer compatible with the current stable version. If you want to take a look at, or test, Prometheus, you will still want to go for this version: the legacy version will disappear in the foreseeable future.

As an interface to Prometheus, the Alert Manager has its own API. When the Alert Manager receives an alert from Prometheus, it can process that alert in a variety of ways: the manager can mail a warning or forward the alert to the Pager Duty API.

The pager, in turn, supports various modules. Out of the box are Pager Duty modules for Slack, Hipchat, and various other chat protocols. But you could hitch up Pager Duty to a text service with a matching SMS module.

PromDash

PromDash (Figure 5) is a powerful tool for rendering the metrics stored in Prometheus in a meaningful way. The tool's developers describe PromDash as a dashboard builder, thus setting it apart from standard dashboards.

Figure 5: Prom Dash is a helper that lets administrators create graphs based on Prometheus data.

PromDash does not try to give users prebuilt templates for charts but instead gives administrators a tool for creating these templates themselves. PromDash can thus rightly be described as a graphical helper that facilitates the use of the PromQL language.

After setting up the solution, you need to add the servers that will act as data sources in PromDash, then sit back and enjoy the ride. One of PromDash's features is the ability to annotate graphs or integrate them directly into other pages. For example, you could easily implement a status dashboard for the entire platform that would give you a centralized overview of all the systems you need to monitor.

The solution for configuring PromDash itself is also very smart: The PromDash module stores the configured dashboards in a MySQL database in the background. This design means the configuration and program are separate, so you can run any number of PromDash instances on the same host without you needing to manually synchronize the configurations.