Building a Web Spider with Ruby

Spider on the Web

Article from Issue 51/2005
Author(s):

Ruby is a very elegant language,and it’s harmonious – the parts work together effectively. Ruby also significantly reduces a developer’s burden. We’ll show you how to use Ruby to build a quick and simple web spider application.

Ruby is a scripting language developed by Yukihiro Matsumoto and released under the GPL. The Ruby language has an excellent set of string manipulation and networking libraries, making it a great choice for writing web spiders. If you are not familiar with web spiders, they are programs designed to automatically traverse the web. Search engines use web spiders to add web pages to their index; companies like Netcraft use spiders to get statistics on web servers. You can use a web spider to find information automatically from almost any website; in this article, we’ll discuss how to use Ruby to retrieve information from LiveJournal, a popular weblog provider. You can extend these techniques to virtually every website that provides public information.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
Print Issues
Digital Issues
 
SUBSCRIPTIONS
Print Subs
Digisubs
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Portia

    Are you interested in retrieving stock quotes in machine-readable form off the Internet? No problem: After a few mouse clicks, Portia weaves a command line and wraps the data in JSON format.

    more »
  • Security Lessons – Ruby Tools

    Great tools and resources are available to help you write secure Ruby on Rails code. Kurt examines some tools and offers some tips.

    more »
  • Rails Framework

    Most web libraries make 90 percent of the job simple and the rest impossible. Rails,an open source framework programmed in Ruby,is flexible enough to succeed with that remaining 10 percent.

    more »
  • Books

     

    more »
  • Ruby and Glade

    Application development shouldn’t be a chore. We’ll show you how to simplify the development process with the Ruby programming language and the Glade interface design utility.

    more »
comments powered by Disqus

Visit Our Shop

Trial Subscription

Digital Subscription

Subscribe

Direct Download

Read full article as PDF:

Ruby_Web_Spiders.pdf (165.09 kB)

News

Tag Cloud

Administration Community Events Hardware Linux Linux New Media Linux Pro Magazine Mobile Programming Software Ubuntu Web Development Windows free software open source