Simple web scraping with Bash

Ski Report

© Photo by Nicolai Berntsen on Unsplash

© Photo by Nicolai Berntsen on Unsplash

Article from Issue 262/2022
Author(s):

With one line of Bash code, Pete scrapes the web and builds a desktop notification app to get the daily snow report.

While recently doing a small project, I was amazed by how much web scraping I could do with just one line of Bash. I used the text-based Lynx browser [1] and then piped the output to a grep search. Figure 1 shows the one-line Bash example that scrapes the current snow depth from the Sunshine Village Snow Forecast web page.

In this article, I will introduce some techniques to easily scrape web pages, and then I will create a desktop notification script that provides the daily snow forecast.

The Lynx Text Browser

For my Bash web scraping, I started out by looking at using command-line tools such as curl [2] with the html2text [3] utility. This technique definitely works, but I found that using the Lynx browser offers a one-step solution with a slightly cleaner text output.

[...]

Use Express-Checkout link below to read the full article (PDF).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Bash Tuning

    In the old days, shells were capable of little more than calling external programs and executing basic, internal commands. With all the bells and whistles in the latest versions of Bash, however, you hardly need the support of external tools.

  • Bash Web Server

    With one line of Bash code, you can create a Bash web server for quickly viewing the output from Bash scripts and commands.

  • Shell Download Manager

    A few lines of shell code and the Gawk scripting language make downloading files off the web a breeze.

  • INSECURITY NEWS / USERFRIENDLY
  • Bash vs. Vista PowerShell

    Microsoft’s new PowerShell relies on .NET framework libraries and thus has access to a treasure trove of functions and objects. How does PowerShell measure up to traditional shells like Bash?

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News