Home network monitoring with pfSense, Protectli, and a screen scraper
Programming Snapshot – Protectli

What is making the lights on the router flicker so excitedly? An intruder? We investigate with pfSense on a Protectli micro appliance and a screen scraper to email the information.
It's a shame that no routers simply display the network packet addresses that pass through them on an LED display. Because I'm curious about what's going on in my home network, on the advice of a work colleague, I bought a micro appliance from the Chinese company Protectli (Figure 1) [1], which runs the FreeBSD-based open source firewall pfSense. The box is about four by four inches in size and passively cooled, so there is absolutely no fan or other noise.

The installation is a piece of cake – simply load the distribution from the pfSense Community Edition website [2] onto a bootable USB stick, insert an mSATA disk and RAM into the Protectli's small case, and after booting, say yes to the installation prompts. Badda-bing badda-boom, pfSense's web GUI is up and running (Figure 2).

Guardian at the Gate
The Protectli appliance is directly connected to the Internet-facing interface (in my case, a DSL modem to the ISP). On the LAN side, it provides access to the Internet for all devices connected to my home network (in my case, a series of routed subnets).
Equipped with a four-core Celeron, it's powerful enough to look at every single packet, create statistics or even intervene when needed, and block certain communication attempts according to predefined firewall rules. If I want to know why router lights are flickering, I only need to call the pfSense GUI to see who is streaming Spotify, watching Netflix, or ordering on Amazon (Figure 2).
In addition to traditional terminal-based tools such as pftop
, the firewall GUI also offers very elegant add-on packages such as ntopng
, so you can browse through pie charts and HTML tables to find out who uses the most bandwidth or contacts computers in dubious countries (Figure 3).

Unfortunately, there is no official API for the GUI, only a FauxAPI [3], which runs as an add-on package on the pfSense distribution and provides limited access to the firewall's internals.
Keys for Protectli
To check at regular intervals what is happening on the Protectli box, I thought it would be easy to write a screen scraper [4] that logs in periodically and automatically to the box's login page (Figure 4) and scans and mails the data displayed on the dashboard.

However, the first hurdle between a command-line client and the juicy network data is the login page so defiantly presented by pfSense. A look at the HTML code (Figure 5) reveals that the two fields for accepting the username and password are dubbed usernamefld
and passwordfld
, and the submit button goes by the name login
. The Python scraper quickly thrown together in Listing 1 [5], which uses the selenium
module installed using pip3
to simulate a browser, searches for and finds these elements using the find_element_by_name()
function.
Listing 1
dash-scraper.py

For the webdriver.Firefox()
call to work with the system's Firefox browser, the Linux distribution needs the geckodriver
program, which is available as a TAR file [6]. You need to unpack this and dump the binary that falls out of the archive into a path that can be found somewhere in $PATH
. The script opens the browser, takes you to the login page, autofills the form fields, and then clicks on the Login button. The selenium
module is often used for testing WebGUIs and makes it really easy to simulate a user sitting in front of a web user interface.
After the pfSense login screen, the pfSense dashboard page (Figure 6) with the firewall overview data is saved in the saved.png
file by the new screen scraper by calling save_screenshot()
as a trigger. The values to be filled in are read by the script from the creds.yaml
file, which is read from disk; the data is then stored as a username and password accessible in the creds
dictionary (Figure 7).


One Man Went to Mow
Listing 2 is used to periodically send the collected data to an email address. It bundles the PNG file created by Listing 1 into an email as an attachment and sends it via an SMTP server. To read the security-relevant username and password variables for the SMTP server, line 10 retrieves the same YAML file as before and stores its contents in the creds
dictionary.
Listing 2
mail.py
The script then builds an HTML body – with introductory text and an IMG link to the attached image, so that a webmail client can display it graphically. As of line 31, Listing 2 establishes a connection to the SMTP server, whose address is also retrieved from the creds.yaml
file as the smtp_server: mail.provider.net
entry. The script uses port 587 and transmits the data in TLS-encrypted form. It appends the screenshot in MIME format and adds an email content ID header with the name of an imaginary file in square brackets.
Figure 8 shows how the mail arrives in Gmail. Called as a cron job once a day, this keeps the home owner up to date with what's happening on the local network.

Infos
- Protectli micro appliance: https://www.amazon.com/gp/product/B01GIVQI3M
- pfSense Community Edition download: https://www.pfsense.org/download/
- pfSense FauxAPI: https://github.com/ndejong/pfsense_fauxapi
- Jarmul, Katharine, and Richard Lawson. Python Web Scraping, 2nd ed.Packt Publishing, 2017
- Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/208/
- geckodriver: https://github.com/mozilla/geckodriver/releases/tag/v0.19.0
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
SparkyLinux 6.6 Now Available for Installation
The Debian-based SparkyLinux has a new point release that retools the live USB desktop creator and other changes that give it shiny new-ness.
-
SparkyLinux 6.6 Now Available for Installation
The Debian-based SparkyLinux has a new point release that retools the live USB desktop creator and other changes that give it shiny new-ness.
-
Escuelas Linux 8.0 Now Available
Just in time for its 25th anniversary, the developers of Escuelas Linux have released the latest version.
-
LibreOffice 7.5 Loaded with New Features and Improvements
The favorite office suite of the Linux community has a new release that includes some visual refreshing and new features across all modules.
-
The Next Major Release of Elementary OS Has Arrived
It's been over a year since the developers of elementary OS released version 6.1 (Jólnir) but they've finally made their latest release (Horus) available with a renewed focus on the user.
-
KDE Plasma 5.27 Beta Is Ready for Testing
The latest beta iteration of the KDE Plasma desktop is now available and includes some important additions and fixes.
-
Netrunner OS 23 Is Now Available
The latest version of this Linux distribution is now based on Debian Bullseye and is ready for installation and finally hits the KDE 5.20 branch of the desktop.
-
New Linux Distribution Built for Gamers
With a Gnome desktop that offers different layouts and a custom kernel, PikaOS is a great option for gamers of all types.
-
System76 Beefs Up Popular Pangolin Laptop
The darling of open-source-powered laptops and desktops will soon drop a new AMD Ryzen 7-powered version of their popular Pangolin laptop.
-
Nobara Project Is a Modified Version of Fedora with User-Friendly Fixes
If you're looking for a version of Fedora that includes third-party and proprietary packages, look no further than the Nobara Project.