Building a Web Spider with Ruby
Spider on the Web
Ruby is a very elegant language,and it’s harmonious – the parts work together effectively. Ruby also significantly reduces a developer’s burden. We’ll show you how to use Ruby to build a quick and simple web spider application.
Ruby is a scripting language developed by Yukihiro Matsumoto and released under the GPL. The Ruby language has an excellent set of string manipulation and networking libraries, making it a great choice for writing web spiders. If you are not familiar with web spiders, they are programs designed to automatically traverse the web. Search engines use web spiders to add web pages to their index; companies like Netcraft use spiders to get statistics on web servers. You can use a web spider to find information automatically from almost any website; in this article, we’ll discuss how to use Ruby to retrieve information from LiveJournal, a popular weblog provider. You can extend these techniques to virtually every website that provides public information.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
US / Canada
UK / Australia
News
-
Cisco Releases Annual Security Report
Spammers go low-volume, and 90% of IE browsers are unpatched.
-
Zero Day Exploits Target Flash
Adobe scrambles to release patches for vulnerable Flash Player.
-
Intel Unveils Compute Stick
Four-inch-long computer on a stick lets you boot a full Linux system from any HDMI display device.
-
Obama Proposes Personal Data Notification Law
New statute would require companies to report break-ins to consumers.
-
TGXf Project Warns of File Transfer through Screen Pixels
Weird data transfer technique avoids all standard security measures.
-
Industry Giants Announce a Fix for the Password Mess
FIDO alliance declares the beginning of the end for old-style login authentication.
-
Poll: Vote for the Best Open Source Software for Photographers
The Linux New Media Awards have honored the most significant products, projects, people, and organizations for open source/Linux every year since 2000.
-
Debian Gets Forked
Legendary Uber-distro splits over the systemd controversy.
-
Linux Mint Project Releases Mint 17.1
New LTS version offers many refinements for the Cinnamon and Mate desktops and significant improvement under the hood.
-
CeBIT Open Source Forum Call for Papers
One of CeBIT’s most successful forums returns in 2015.