Benchmark sparks

Dear Linux Pro Reader,

"Figures don't lie, but liars figure," they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election. At this writing, two candidates are both claiming they are ahead in the race and quoting different polls and surveys. To each, the other's polls and surveys don't even exist – their world does not have room for information that conflicts with their story.

This same phenomenon was also in the High Tech news this month, when AMD posted a video on YouTube criticizing Intel's use of the SYSmark benchmarking tool for measuring PC performance. SYSmark shows the latest Intel processors outperforming AMD equivalents by a large margin. In the video, AMD spokesmen John Hampton and Tony Salinas argued that the alternative PCMark  8 benchmark shows a much more competitive race between the Intel and AMD chips, and they demonstrate some practical use cases that show the PCMark  8 is more accurate than anything Intel is using.

They even quote a 2010 Federal Trade Commission (FTC) ruling that forced Intel to include some fine print with their SYSmark citations stating, "Software and workloads in performance tests may have been optimized only on Intel microprocessors."

The problem, according to Hampton and Salinas, is that SYSmark places the emphasis on the CPU only, whereas PCMark  8 looks at the CPU, GPU, and video subcomponents together, which is more like how the computer actually behaves.

I try not to take sides in vendor disputes, but in this case, since the FTC has taken sides, it does seem very likely that AMD has a point. If SYSmark is not an accurate benchmark for comparing real-world PC system performance, why does Intel keep using it? Because marketing departments of major corporations don't exactly put the emphasis accuracy.

Hampton invokes another recent news story when he mentions "the recent debacle over the emissions standards provided by a major auto maker …" This apparent reference to the Volkswagen emissions scandal highlights another case where customers were misled with numbers. Of course, as Intel would readily point out, Volkswagen is actually providing inaccurate information, whereas Intel is reporting the results of the SYSmark benchmarks very accurately. Is using the wrong facts as bad as giving the wrong answer? If it leads people to incorrect conclusions, the effect is the same.

We get so many numbers thrown at us every day that we really can't chase down all of them. Still, if you're going to go to the trouble of buying a computer that you will have to work with every day for at least a couple of years, it really might be worth taking a closer look at the numbers. The next time I read a triumphant claim of success that references a computational benchmark, I'm going to Google the benchmark and find out what it is really testing.