Home network monitoring with pfSense, Protectli, and a screen scraper
Programming Snapshot – Protectli
What is making the lights on the router flicker so excitedly? An intruder? We investigate with pfSense on a Protectli micro appliance and a screen scraper to email the information.
It's a shame that no routers simply display the network packet addresses that pass through them on an LED display. Because I'm curious about what's going on in my home network, on the advice of a work colleague, I bought a micro appliance from the Chinese company Protectli (Figure 1) [1], which runs the FreeBSD-based open source firewall pfSense. The box is about four by four inches in size and passively cooled, so there is absolutely no fan or other noise.
The installation is a piece of cake – simply load the distribution from the pfSense Community Edition website [2] onto a bootable USB stick, insert an mSATA disk and RAM into the Protectli's small case, and after booting, say yes to the installation prompts. Badda-bing badda-boom, pfSense's web GUI is up and running (Figure 2).
Guardian at the Gate
The Protectli appliance is directly connected to the Internet-facing interface (in my case, a DSL modem to the ISP). On the LAN side, it provides access to the Internet for all devices connected to my home network (in my case, a series of routed subnets).
Equipped with a four-core Celeron, it's powerful enough to look at every single packet, create statistics or even intervene when needed, and block certain communication attempts according to predefined firewall rules. If I want to know why router lights are flickering, I only need to call the pfSense GUI to see who is streaming Spotify, watching Netflix, or ordering on Amazon (Figure 2).
In addition to traditional terminal-based tools such as pftop
, the firewall GUI also offers very elegant add-on packages such as ntopng
, so you can browse through pie charts and HTML tables to find out who uses the most bandwidth or contacts computers in dubious countries (Figure 3).
Unfortunately, there is no official API for the GUI, only a FauxAPI [3], which runs as an add-on package on the pfSense distribution and provides limited access to the firewall's internals.
Keys for Protectli
To check at regular intervals what is happening on the Protectli box, I thought it would be easy to write a screen scraper [4] that logs in periodically and automatically to the box's login page (Figure 4) and scans and mails the data displayed on the dashboard.
However, the first hurdle between a command-line client and the juicy network data is the login page so defiantly presented by pfSense. A look at the HTML code (Figure 5) reveals that the two fields for accepting the username and password are dubbed usernamefld
and passwordfld
, and the submit button goes by the name login
. The Python scraper quickly thrown together in Listing 1 [5], which uses the selenium
module installed using pip3
to simulate a browser, searches for and finds these elements using the find_element_by_name()
function.
Listing 1
dash-scraper.py
For the webdriver.Firefox()
call to work with the system's Firefox browser, the Linux distribution needs the geckodriver
program, which is available as a TAR file [6]. You need to unpack this and dump the binary that falls out of the archive into a path that can be found somewhere in $PATH
. The script opens the browser, takes you to the login page, autofills the form fields, and then clicks on the Login button. The selenium
module is often used for testing WebGUIs and makes it really easy to simulate a user sitting in front of a web user interface.
After the pfSense login screen, the pfSense dashboard page (Figure 6) with the firewall overview data is saved in the saved.png
file by the new screen scraper by calling save_screenshot()
as a trigger. The values to be filled in are read by the script from the creds.yaml
file, which is read from disk; the data is then stored as a username and password accessible in the creds
dictionary (Figure 7).
One Man Went to Mow
Listing 2 is used to periodically send the collected data to an email address. It bundles the PNG file created by Listing 1 into an email as an attachment and sends it via an SMTP server. To read the security-relevant username and password variables for the SMTP server, line 10 retrieves the same YAML file as before and stores its contents in the creds
dictionary.
Listing 2
mail.py
The script then builds an HTML body – with introductory text and an IMG link to the attached image, so that a webmail client can display it graphically. As of line 31, Listing 2 establishes a connection to the SMTP server, whose address is also retrieved from the creds.yaml
file as the smtp_server: mail.provider.net
entry. The script uses port 587 and transmits the data in TLS-encrypted form. It appends the screenshot in MIME format and adds an email content ID header with the name of an imaginary file in square brackets.
Figure 8 shows how the mail arrives in Gmail. Called as a cron job once a day, this keeps the home owner up to date with what's happening on the local network.
Infos
- Protectli micro appliance: https://www.amazon.com/gp/product/B01GIVQI3M
- pfSense Community Edition download: https://www.pfsense.org/download/
- pfSense FauxAPI: https://github.com/ndejong/pfsense_fauxapi
- Jarmul, Katharine, and Richard Lawson. Python Web Scraping, 2nd ed.Packt Publishing, 2017
- Listings for this article: ftp://ftp.linux-magazine.com/pub/listings/linux-magazine.com/208/
- geckodriver: https://github.com/mozilla/geckodriver/releases/tag/v0.19.0
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.
-
Valve and Arch Linux Announce Collaboration
Valve and Arch have come together for two projects that will have a serious impact on the Linux distribution.
-
Hacker Successfully Runs Linux on a CPU from the Early ‘70s
From the office of "Look what I can do," Dmitry Grinberg was able to get Linux running on a processor that was created in 1971.
-
OSI and LPI Form Strategic Alliance
With a goal of strengthening Linux and open source communities, this new alliance aims to nurture the growth of more highly skilled professionals.
-
Fedora 41 Beta Available with Some Interesting Additions
If you're a Fedora fan, you'll be excited to hear the beta version of the latest release is now available for testing and includes plenty of updates.
-
AlmaLinux Unveils New Hardware Certification Process
The AlmaLinux Hardware Certification Program run by the Certification Special Interest Group (SIG) aims to ensure seamless compatibility between AlmaLinux and a wide range of hardware configurations.
-
Wind River Introduces eLxr Pro Linux Solution
eLxr Pro offers an end-to-end Linux solution backed by expert commercial support.