Aggregating data with Portia

The Internet is a treasure trove of useful information, often residing on colorful HTML pages that are not easily extracted and processed. If you want to automate processing of current stock quotes or aggregate news, for example, you need to dismantle the HTML code of news portals such as CNN or Slashdot. This can be pretty ugly work.

Portia, a tool written in Python [1], promises a remedy; its name also refers to a genus of spiders, which would seem to make sense on the World Wide Web. The tool consists of a web application that, with a simple click, allows a user to select stock quotes, messages, and any other desired content. Portia then extracts this data and outputs it in JSON format.

Supported by a supplied web crawler, Portia can also ransack complete websites. As an example, if you need the headings from all Wikipedia articles, you show Portia exactly once where the headline resides on a Wikipedia page. The crawler then traverses the entire website and returns all matching headings in JSON format (see the "Warning" box for more information).

[...]

Use Express-Checkout link below to read the full article (PDF).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
Print Issues
Digital Issues
 
SUBSCRIPTIONS
Print Subs
Digisubs
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • SpiderOak

    Back up, synchronize, version, and collaborate with SpiderOak.

    more »
  • Spider, Spider

    Scrapy is an open source framework written in Python that lets you build your own crawlers with minimal effort for professional results.

    more »
  • Ruby Web Spiders

    Ruby is a very elegant language,and it’s harmonious – the parts work together effectively. Ruby also significantly reduces a developer’s burden. We’ll show you how to use Ruby to build a quick and simple web spider application.

    more »
  • Shell Download Manager

    A few lines of shell code and the Gawk scripting language make downloading files off the web a breeze.

    more »
  • Welcome

    Sometimes you get a reminder of something you already know, and you have to stop and say, "Ah, yes, that's right, I'm awake." Several news sources ran a story recently on Google's Safe Browsing technology, which "scan websites for potential risks to warn users before they visit unsafe sites." Safe browsing is now integrated into the Chrome and Firefox browsers, which means users get a warning about sites with potentially unsafe content.

    more »
comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News

Tag Cloud

Administration Community Events Hardware Linux Linux Pro Magazine Mobile Programming Security Software Ubuntu Web Development Windows free software open source