Off the Beat: Bruce Byfield's Blog

Reviewing the latest Linux Mint release, I discovered the DuckDuckGo search engine. Linux Mint is using DuckDuckGo as its default search engine in a revenue sharing plan, and, given DuckDuckGo's friendliness to free software, as well its privacy tools, the choice is one that should appeal to many. But how does DuckDuckGo's search results compare to that of the search giants Google and Bing?

I'm still working out a detailed answer to that question. For one thing, DuckDuckGo is a hybrid search engine, drawing answers from over fifty sources, so probably the answers vary. For another, when I use Google, as a Canadian I am redirected towards the Canadian versions of the site, which might give slightly different results from the main American site. In addition, search results are very much what you make them, and comparing basic searches may not give a completely accurate impression of what each engine is capable of.

But, to provide an interim answer, I compared the first ten results on three search terms on which I am confident that I have a high-level of expertise: "bruce byfield," "nanday" (a small South American parrot, of which I currently keep three), and "kde." I chose the last two to include an obscure and well-known topic, respectively. The results suggest not only that DuckDuckGo can hold its own against the giants, but that search results in general should be treated with caution.

Search #1: "bruce byfield"

Bing chose to lead with my personal blog, then the entry on me in Wikipedia. It didn't notice that my blog biography was more or less identical to the Wikipedia entry, or that another biography was eight years old. It also included a selection of articles on the topics of OpenOffice.org and MS Office, Ubuntu's year old Lucid Linx, cusotmization tools in GNOME, KDE, and Unity, Mono, and buying an HP laptop five years ago. The selection of articles indicates something of my range, but the biographical selections seemed rather poor.

DuckDuckGO begins with its one-click answer, which summarizes me in a sentence, and directs readers to my Wikipedia entry. It also lists entries on Facebook, LinkedIn, Flickr, and my personal blog and website. It includes one entry that applies to someone who shares my name, and also my long-neglected Linux Journal and Classmates profile, and a bio that cribbed from my Wikipedia entry. A comparison of OpenOffice.org and MS Office and a video on the same topic rounded off the results. Despite several poor sources, these results gave a reason impression of me (always assuming, of course, that anyone would care).

Google began with my Google Plus Profile, immediately showing its bias. Another possible bias is suggested by placing my lightly tended website second -- an assumption that may have been reasonable once, but is hardly so these days when personal websites have given way to blogs and social media profiles. Google also included my Linux Journal profile. It did better by including my latest personal and Linux Pro Magazine blog, but also included two year old stories, "The Mono Mystery That Wasn't" and "Writing About FOSS sexism," both of which are on popular topics, but hardly give a sense of the sorts of things I generally write about.

Verdict: Despite the false positive and one questionable source, DuckDuckGo did the best job of rounding up biographical details. By contrast, Google did the best job of reflecting what I've been writing recently, but its selection of biographical results was poor. Bing's results were far behind either DuckDuckGo or Google. All three would be improved by the ability to weed out repetition, and by giving less weight to results older than three years.

Search #2: "nanday"

Naturally enough, Bing offered the nanday.com site first, and, further down the list, included the same site's photo gallery. The only trouble is, nanday.com is primarily a mailing list, which doesn't have the sort of overview that most searchers would probably want. It also included a Facebook page for someone with the first name of "Nanday." It was on stronger grounds with the Wikipedia entry, and also included the Wikipedia entry on "conures," a classification that includes nandays. Its other entries were encylopedia-like entries, mostly from pet sites, with information that was often obsolete. Users of these results would be left in no doubt about what a nanday looks like, but would have only a handful of facts -- some of which are questionable.

Cribbing from Wikipedia, DuckDuckGo gives a one-click answer describing what nanday look like. Then, like Bing, DuckDuckGo began with nanday.com and Wikipedia, and continued with encyclopedia-like entries from a variety of sites. The main difference from Bing's results, is that some different but comparable sites appeared, and the final result, the International Conure Association was more reputable than any of Bing's results. However, I still wouldn't consider the results very reliable.

Google liked nanday.com so well that it features in its first two results. A selection of images followed, with the Wikipedia entry fourth. Two Youtube videos and a similar selection of questionably accurate encyclopedia articles followed.

Verdict: None of the search sites did well on this admittedly obscure topic. At best, they provide a basic overview and visual identification.

Search #3: "kde"

Bing's results give a strong impression of KDE and the software and community associated with it. They start with the main site, kde.org, then Wikipedia and KDE News (also kown as The Dot). The KDE Techbase, KDE Windows Projbect, KDE Women, KDE Software Compilation, and KDE Education sites are also mentioned, along with the community wiki. A mailing list thread comparing KDE and GNOME rounds off the results.

DuckDuckGo's results are good, but somewhat weaker than Bing's. After mentioning kde.org and Wikipedia, DuckDuckGo mentions some of the same resources, but in different order -- most significantly, placing KDE News tenth instead of third. It also mentions two KDE-related projects, the Kopete chat tool and the Amarok music manager, and a false positive of the Kentucky Department of Education.

Google begins with kde.org, but sub-divided into another half a dozen links, most of which Bing and DuckDuckGO offer seperates. After returning Wikpedia, the reuslts veer into some questionable places, such as KDE on identi.ca, the KDE Fedora spin, KOffice, and KDE on Cygwin. Usefully, it mentions the KDE-look and PlanetKDE sites, but many of its references are of dubious value.

Verdict: Bing's results are the most useful, with DuckDuckGo's and Googles adequate, but somewhat less so.

A few observations

If these results are typical, then DuckDuckGo compares well with the search giants. In two of these tests, it outperformed Bing, and in all of them it was better or at least comparable to Google. Since DuckDuckGo's sources include both Bing and Google, these results might seem predictable in retrospect, but what I didn't apprciate before doing the exercise is the extent to which DuckDuckGo's ranking of results can differ.

Even more importantly, although I do dozens of searches a day, I didn't fully appreciate the limits and biases of the results I rely on. If I can find problems with the results for subjects for which I have expertise, what other problems lurk in results that I can't judge because I am a novice? The most I can say is that DuckDuckGo's one-click answers provide the quick answers that is all many searchers might want. But even DuckDuckGo includes some eccentricities, although none (so far as I can see) so obviously biased as including results from its own services ahead of anything else.

My tentative conclusion is that DuckDuckGo delivers results that are comparable to Google's and Bing's, but not obviously or consistently superior. As I've mentioned, there are other reasons for preferring DuckDuckGo, but the search results don't appear to be among them.

But, then, I've only looked closely at three results. If anyone else wants to compare results on the three search engines, I'd be interested in hearing how their experience compares to mine.