Screen scraping with Colly in Go
Programming Snapshot – Colly
Lead Image © Hannu Viitanen, 123RF.com
The Colly scraper helps developers who work with the Go programming language to collect data off the web. Mike Schilli illustrates the capabilities of this powerful tool with a few practical examples.
As long as there are websites to view for the masses of browser customers on the web, there will also be individuals on the consumer side who want the data in a different format and write scraper scripts to automatically extract the data to fit their needs.
Many sites do not like the idea of users scraping their data. Check the website's terms of service for more information, and be aware of the copyright laws for your jurisdiction. In general, as long as the scrapers do not republish or commercially exploit the data, or bombard the website too overtly with their requests, nobody is likely to get too upset about it.
Different languages offer different tools for this. Perl aficionados will probably appreciate the qualities of WWW::Mechanize as a scraping tool, while Python fans might prefer the selenium package [1]. In Go, there are several projects dedicated to scraping that attempt to woo developers.
[...]
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
US / Canada
UK / Australia