Benchmark sparks
Benchmark sparks
“Figures don’t lie, but liars figure,” they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election.
Dear Linux Pro Reader,
"Figures don't lie, but liars figure," they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election. At this writing, two candidates are both claiming they are ahead in the race and quoting different polls and surveys. To each, the other's polls and surveys don't even exist – their world does not have room for information that conflicts with their story.
This same phenomenon was also in the High Tech news this month, when AMD posted a video on YouTube criticizing Intel's use of the SYSmark benchmarking tool for measuring PC performance. SYSmark shows the latest Intel processors outperforming AMD equivalents by a large margin. In the video, AMD spokesmen John Hampton and Tony Salinas argued that the alternative PCMark 8 benchmark shows a much more competitive race between the Intel and AMD chips, and they demonstrate some practical use cases that show the PCMark 8 is more accurate than anything Intel is using.
They even quote a 2010 Federal Trade Commission (FTC) ruling that forced Intel to include some fine print with their SYSmark citations stating, "Software and workloads in performance tests may have been optimized only on Intel microprocessors."
The problem, according to Hampton and Salinas, is that SYSmark places the emphasis on the CPU only, whereas PCMark 8 looks at the CPU, GPU, and video subcomponents together, which is more like how the computer actually behaves.
I try not to take sides in vendor disputes, but in this case, since the FTC has taken sides, it does seem very likely that AMD has a point. If SYSmark is not an accurate benchmark for comparing real-world PC system performance, why does Intel keep using it? Because marketing departments of major corporations don't exactly put the emphasis accuracy.
Hampton invokes another recent news story when he mentions "the recent debacle over the emissions standards provided by a major auto maker …" This apparent reference to the Volkswagen emissions scandal highlights another case where customers were misled with numbers. Of course, as Intel would readily point out, Volkswagen is actually providing inaccurate information, whereas Intel is reporting the results of the SYSmark benchmarks very accurately. Is using the wrong facts as bad as giving the wrong answer? If it leads people to incorrect conclusions, the effect is the same.
We get so many numbers thrown at us every day that we really can't chase down all of them. Still, if you're going to go to the trouble of buying a computer that you will have to work with every day for at least a couple of years, it really might be worth taking a closer look at the numbers. The next time I read a triumphant claim of success that references a computational benchmark, I'm going to Google the benchmark and find out what it is really testing.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.