Build your own crawlers
Spider, Spider
© Lead Image © mtkang, 123RF.com
Scrapy is an open source framework written in Python that lets you build your own crawlers with minimal effort for professional results.
A crawler demonstrates the capabilities of version 1.0 of the Scrapy framework [1] running under Python 2.7 [2]. Scrapy is an open source framework for extracting data from websites. It recursively crawls through HTML documents and follows all the links it finds.
In the spirit of HTML5, the test created in this article is designed to reveal non-semantic markup on websites. The crawler counts the number of words used per page, as well as the number of characteristic tag groups (Table 1), saving the results along with the URL in a database.
To install the required packages, I used the Debian 8 Apt package manager:
[...]
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Bottlenecks Discovered by Meta Engineers Lead to Scalability Improvements
Thanks to a small group of engineers at Meta, the Linux kernel module kernfs has been improved, which has led to some serious gains.
-
PwnEscape Poses VM Vulnerability
A guest-to-host escape flaw in Linux KVM, named PwnEscape, might be yet another means of attacking Linux.
-
Linux Mint 27 Set for a Christmas Release
The developers behind Linux Mint have announced a Christmas release, and it looks like Santa's sack will be brimming with features.
-
Is Linus Torvalds Vibe Coding?
The creator of Linux has added AI into his process for Linux kernel development, and it's more than just for debugging.
-
openKylin 3.0 Released with Expanded AI Integration
The latest release features a pre-installed AI agent to help perform tasks.
-
DigitalOcean Ends Open Source Credits Program
In a change that has surprised many open source developers, DigitalOcean has decided to sunset its Open Source Credits Program.
-
Office 365 Available for Testing on Linux via Bottles
Linux users can now test the official Office 365, thanks to Mirko Brombin's work on Bottles.
-
Dutch Government Turns to Open Source for Digital Sovereignty
Following in the footsteps of other European countries, the Netherlands is looking to increase its sovereign digital infrastructure with open source software.
-
Linux May Soon Work on Snapdragon X2-Based PCs
If you're thinking about buying a Snapdragon X2-based desktop or laptop, Linux may soon be an option.
-
KDE for People Initiative Calls for AI Ban in Plasma
A new community of KDE wants the project to adopt a strict no-AI policy for KDE Plasma.