Off the Beat: Bruce Byfield's Blog

In October 2011, I wrote about Felipe Contreras' attempt to poll GNOME users, and the resistance he met. Ten months later, Conteras has only crunched about 20% of the replies, but has seen enough of the rest to suggest that they would not greatly change the results. And guess what? The survey seems to have produced reasonably useful and unbiased results.

If you remember, Felipe Contreras claimed last year that his attempts to interest the GNOME project in a poll caused GNOME developers to attack both him and his proposal. Many insisted that no online poll could ever claim accuracy, while others claimed than only those already biased against the GNOME 3 release series would bother to participate. The fact that  Contreras often responded to this obstruction angrily did little to help his idea. In the end, his poll was only launched because Michael Larabel posted it on Phoronix.

The Problem of Bias

Theoretically, Contreras' critics were right. A self-selected poll has problems with being a representative sample that can only be partially overcome, and even then only with great difficulty. However, that is not a reason for rejecting the entire idea of the poll. For one thing, those answering may be homogenous enough that the question of obtaining a representative sample may be largely irrelevant; it may depend on the interest groups identified.

Rather than reason to demand an impossible perfection, the possibility of bias is more a caution to treat the results as only broad indicators. Lopsided results should be treated cautiously, if not rejected outright, while results within a few percent of each other have to be treated as equivalent for caution's sake. 

For the most part, this is how Contreras treats the data, although he slips at least once, when he apparently generalizes about the opinons of those who don't use terminals from three responses. Generally, however, he remembers to treat the poll as a very inexact instrument, and remains careful not to claim too much. In fact, if anything, he could make much stronger claims if he didn't seem determined to be as reasonable-sounding as possible.

In addition, Contreras makes some effort to address the problem of bias, although he seems aware that it can never be completely overcome. Including in the poll the question, "How are you taking this survey?" he finds that those who selected "Somebody is pushing me to do it," did not differ much from those who selected "Completely on my own." He also finds that all participants avoided extremes when giving answers about what they thought of GNOME. He concludes that " GNOME people were wrong; there’s no bias." 

This conclusion can be questioned if you insist that people would lie about why they were taking the survey. However, it could just as plausibly be argued that GNOME advocates would make a point of taking the survey just as much as GNOME flamers would. Under these circumstances, the worst you can say about Contreras' conclusion is that it is too emphatic -- it would be more accurate to say that any bias was too low to be detected with the methodology used.

The Results

Contreras's first finding was that about 45% said that GNOME had become worse in the past year -- that is, since GNOME 3 was released -- and just over 40% said it had become worse. He summarizes this result by saying that opinion was "50/50". What he does not say, I will: This seems a high number who see a change for the worse. Users usually have strong preferences in desktops, but until the current age of user revolts, how many would go to the trouble of saying that a desktop that wasn't to their preference had become worse? A few years ago, the most that average users  probably would have said is that they didn't use it, but preferences differ. In other words, the strength of opinion strikes me as a clue that something unusual has been happening, and not the usual disgruntlement that GNOME officially believes is all that is being expressed..

Despite this first finding, most people said they were "mostly" or "halfway" satisifed with GNOME, although more replied "Not At All" than "Completely." Contrary to what some critics of the poll predicted, these answers were not affected by how often someone used a terminal -- a possibility that was probably raised because Phoronix has the reputation of being a highly technical site.

Similarly, satisfaction with GNOME was showed no overall correlation with whether users had contributed to GNOME, or whether they had ever contacted the GNOME project. However, two-thirds of those who had contacted GNOME said that the outcome had been unsatisfactory -- whatever that means. The obvious implication is that GNOME is unfriendly to users or outsiders, although more specific questions would be needed to be sure.

Contreras ends by doing his best to correlate the areas in which people wanted to see improvements. In order, they are:

1. More customization

2. More GNOME 2-like features and design

3. Improved performance and footprint.

4. Nautilus (the file manager)

5. Notifications

6. Adding Restart and Suspend to Shutdown options

7. Improved Theming

8. Better Multiple Monitor Support

9. Improved Evolution (the mail client

10. More Listening to Users

Several of these improvements have already been addressed either by the upcoming GNOME 3.6 or by GNOME Extensions. None seem so unreasonable that they should be rejected out of hand, except possibly the requests for improved performance and footprint, since desktop environments are always been attacked for being bloated.

An Idea Worth Repeating

As Contreras himself is the first to admit, his survey is not perfect. However, although he doesn't always establish his points beyond all doubts, he does manage to make them plausible. Nor have they become any less plausible in light of events and comments made in the year since the poll was taken.

What he has mostly done is given some statistical support for points that observers have been making for some time: that significant numbers prefer GNOME 2, that opinions about GNOME 3 is (or was) sharply divided, and that GNOME is perceived as having an unfriendly attitude towards outsiders.

Just as importantly, he also debunks the comment often heard in the GNOME project that users don't know what they want. Probably, Contreras' summary of wanted improvements was colored partly by the need to interpret answers as they were categorized, but at least five or six of them sound specific enough to give clear direction. The others are more general, but not so much that they fail to indicate the nature of the requested improvements.

So far, I have yet to hear any response from within GNOME. Perhaps there won't be one - which would be unfortunate. In a modest way, Contreras has shown that user polls can produce broadly useful results, and GNOME could do far worse than make a few tweaks and repeat the process annually.

Comments