Building a Web Spider with Ruby
Spider on the Web
Ruby is a very elegant language,and it’s harmonious – the parts work together effectively. Ruby also significantly reduces a developer’s burden. We’ll show you how to use Ruby to build a quick and simple web spider application.
Ruby is a scripting language developed by Yukihiro Matsumoto and released under the GPL. The Ruby language has an excellent set of string manipulation and networking libraries, making it a great choice for writing web spiders. If you are not familiar with web spiders, they are programs designed to automatically traverse the web. Search engines use web spiders to add web pages to their index; companies like Netcraft use spiders to get statistics on web servers. You can use a web spider to find information automatically from almost any website; in this article, we’ll discuss how to use Ruby to retrieve information from LiveJournal, a popular weblog provider. You can extend these techniques to virtually every website that provides public information.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
US / Canada
UK / Australia
Tag Cloud
News
-
Password Management Services Vulnerable to Attack
Should you trust an online service to store your online passwords?
-
New Raspberry Pi Adds Two USB Ports
New B+ board lets you build cool things without the complication of a powered USB hub.
-
Microsoft Grabs No-IP.com Domains
Redmond rushes in to root out alleged malware haven.
-
Firefox Steps into 3D
New initiative will bring futuristic virtual reality effects to the web surfing experience.
-
New Trojan Targets Online Banking
Dyreza malware launches a man-in-the-middle attack that compromises SSL.
-
HP Rolls Out Massive Cloud Network
New cloud combines worldwide access with local attention to data security.
-
New Attack Targets Wireless Logins
A first cousin of the recent Heartbleed attack affects EAP-based wireless and peer-to-peer authentication.
-
Geeks Petition for Free Lenovo BIOS
FOSS community acts to protect freedom of choice for laptop devices.
-
Firefox Says Yes to DRM
Quintessential open source browser shores up its market share with a step toward the proprietary dark side.
-
Webcam Hijackers Busted
Authorities in 16 countries take action against users of the imfamous BlackShades malware tool.