Faster and Higher

The release of Firefox 57 "Quantum" in mid-November caused a stir in the IT community. The Firefox browser was once a state-of-the-art browser, with a huge market share and millions of loyal followers, but over the years, newer and more advanced alternatives have taken away some of the limelight. After years of development and 12 months of re-engineering, the Mozilla developers think they have what they need to get back in the game. The new Firefox is described as twice as fast as the version released a year ago, with a 30 percent savings in memory usage. In addition, Firefox 57 is supposed to be as fast as Google Chrome.

The word on the street is that Firefox is ready for a head-to-head performance comparison. We decided to test Firefox versions 51 and 57 against Chrome 62. We also tested the Opera 49 and Vivaldi 1.12 browsers. Opera, Vivaldi, and Chrome all use the Blink rendering engine – a WebKit fork – so we did not expect any big differences between Blink-based contestants.

We ran the tests on a mid-range Lenovo ThinkPad X220 laptop with 4GB of main memory. The browsers were installed in their default configuration on a newly installed Debian "Sid." We put all test candidates through a series of special browser benchmarks and ran the benchmarks five times on each browser to ensure that no outlier distorted the mean.

The tests used were Speedometer [1], JetStream [2], ARES-6 [3], Kraken [4], WebXPRT [5], MotionMark [6], and HTML5test [7]. These benchmarks are designed to determine the speed at which the browsers process HTML and JavaScript. The WebXPRT benchmark goes even further and also includes image processing and DNA sequencing in the test run. ARES-6 focuses on the speed at which new JavaScript routines are implemented. Except for the ARES-6 and Kraken benchmarks, higher values mean better results.

The HTML5test determines the degree of a browser's compliance with the HTML5 standard. In addition, we looked at the candidates' memory consumption when opening 10 tabs at browser launch time.

Head to Head

At first glance, the figures reveal that Mozilla Firefox 51 was far off the performance of the other browsers – at least in terms of speed. If you have ever run Firefox alongside Chrome, you don't need statistics to confirm that Firefox releases of the recent past have had trouble keeping up.

Firefox 57, however, tells a different story. In some tests, the new Firefox came out ahead of Chrome; in others, it was still behind, but the gap has narrowed. The browsers that use the Blink engine exhibit very similar behavior; sometimes Chrome is slightly in front, sometimes Opera or Vivaldi, sometimes both. For the most part, the variations lie within a range of nuances that play no role in real use.

Mixed Results

We put the browsers through a series of freely available benchmarks. See the box entitled "Browser Benchmarks" for a description of the tests used in this article. In the Speedometer (Figure 1) and ARES-6 (Figure 2) tests, Chrome and its two relatives are clearly in the lead. With WebXPRT (Figure 3) and Kraken (Figure 4), Firefox 57 is ahead, but not as clearly. With its strong focus on graphics performance, the MotionMark test (Figure 5) sees Opera and Vivaldi as winners and Firefox 57 slightly ahead of Chrome. JetStream (Figure 6) ranks Chrome as the winner with Opera in second place.

Figure 1: Speedometer sees the browsers with the Blink engine clearly in front. Firefox 57 closes the gap, doubling the score of Firefox 51.

Figure 2: ARES-6 indicates an advantage for Chrome, Opera, and Vivaldi. With Firefox 51, the test stopped.

Figure 3: In WebXPRT, the most comprehensive test of our course, Firefox 57 is the best. The rest of the field is approximately level, with Firefox 51 trailing behind.

Figure 4: The Kraken test shows a level field, with Firefox 57 in front and Firefox 51 trailing behind.

Figure 5: The results of MotionMark, which tests graphics and animation capabilities, put Opera and Vivaldi on top, followed by Firefox 57 and Chrome.

Figure 6: JetStream shows a clear advantage for Chrome and Opera. Vivaldi and Firefox 57 are approximately equal and outpace Firefox 51.

The HTML5test measures the degree of browser compliance with the HTML5 standard. Of a possible 555 points, the browsers with the Blink engine reach around 520 points, and the two Mozilla browsers lag slightly behind with 465 and 481 points (Figure 7).

Figure 7: Mozilla needs to do some work on W3C HTML5 compliance.

On fairly up-to-date hardware, the detectable speed differences between the browsers are confined to narrow limits. Firefox 57 feels as though it's back in line with Chrome and others, even though it's still slightly behind in some of the measurements.

Mozilla Saves Memory

The situation is completely different when it comes to memory usage. We opened 10 tabs with the same web pages in each of the browsers and then determined memory usage. For Mozilla browsers, you can type about:memory in the address line to check the memory. Click on Measure in the top-left corner; Firefox will calculate the RAM usage and break it down to individual processes. The uppermost displayed value (--explicit) indicates the overall memory usage.

For browsers that use Blink as a web engine, use the Task Manager to determine memory usage. You can open the Task Manager via the More Tools menu or the key combination Shift+Esc. The Task Manager lists the open web pages separately with their memory consumption. The usage of the browser itself and the sandbox with GPU processing are shown separately at the top of the list.

The result of the memory test confirms the widely held belief that Chrome and its descendants are real storage guzzlers. Whereas Chrome, Opera, and Vivaldi use between 337MB and 376MB of memory when opening the 10 web pages, Firefox 51 is content with 249MB, and Firefox 57 occupies just 170MB.

If you work with many open tabs and don't have much RAM, Firefox is definitely better. Computers with 4GB of main memory, which are still quite common, quickly reach their limits with Chrome, Opera, and Vivaldi if you add many open tabs to normal system consumption (Figure 8).

Figure 8: A clear winner in memory usage: Firefox 57 is way ahead, followed by its predecessor in version 51.