Simple web scraping with Bash

Ski Report

© Photo by Nicolai Berntsen on Unsplash

© Photo by Nicolai Berntsen on Unsplash

Article from Issue 262/2022
Author(s):

With one line of Bash code, Pete scrapes the web and builds a desktop notification app to get the daily snow report.

While recently doing a small project, I was amazed by how much web scraping I could do with just one line of Bash. I used the text-based Lynx browser [1] and then piped the output to a grep search. Figure 1 shows the one-line Bash example that scrapes the current snow depth from the Sunshine Village Snow Forecast web page.

Figure 1: One line of Bash code finds the web text for the current snow depth.

In this article, I will introduce some techniques to easily scrape web pages, and then I will create a desktop notification script that provides the daily snow forecast.

[...]

Use Express-Checkout link below to read the full article (PDF).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Web Scraping with Bash

    You can use one line of Bash code to scrape a web page and show the data you're tracking in Home Assistant or Node-RED.

  • Bash Tray Scripts

    YAD lets you customize your system tray with one-line Bash tray scripts.

  • Web Scraping

    Web scraping lets you automatically download and extract data from websites to build your own database. With a simple scraping script, you can harvest information from the web.

  • Xidel

    Xidel lets you easily extract and process data from XML, HTML, and JSON documents.

  • Shell Download Manager

    A few lines of shell code and the Gawk scripting language make downloading files off the web a breeze.

comments powered by Disqus